A poorly designed mod can create a security problem. While you can't prevent hacking entirely, there are some things you should know to make your mod more secure.
Key Point: The primary thing that can make your mod secure is to ensure that all "game logic" (anything that controls what is possible) is processed on the server.
If you're not already aware, Minecraft has a client-server architecture. The game logic happens on the server and the clients basically provide graphical and audio interfaces and accept mouse and keyboard input.
It is possible to make a bad mod design where the client is allowed to tell the server what to do. Let me give you an example.
Imagine you want to make a weapon that has a longer reach than a sword. Because the attack targeting system needs to know what the player is looking at (i.e. where the cross-hairs are, whether it is 3rd person or 1st person view mode, etc.) the client has to be involved in initiating the attack. So let's say you make a client that can check if you are clicking on an entity up to 6 blocks away. That is easy and perfectly fine, but next you have to tell the server what to do with that information. So you need to send a custom packet to the server to tell them that the player did something. There is a wrong way and a right way to handle the packets for this:
- Wrong way: The server receives the packet indicating that the client has detected an attack on an entity that is within six blocks, and so the server goes ahead and processes the attack.
- Right way: The server receives the packet, but before processing the attack, confirms that the targeted entity is within range and that the player is probably looking at it, and then processes the attack.
The first way is wrong, because if the server just acts on the instruction of the client, someone can cheat by making a modified version of your mod that allows them to attack any entity in the game, no matter how close.
The second case is much more secure, because if a hacked client sends a packet to target something that the player can't possibly be attacking, it can be ignored.
Hopefully you understood this important point. To say it another way, the server should always check if any information from the client is valid before acting on it.
- If your mod adds something to the player inventory based on client actions, you should make sure it was possible to acquire.
- If your mod adds achievements based on client side actions, those should be verified on the server side.
- If you created a special food item that gives a variable boost of health, you should make sure that the server determines how much health to add.
Cheating You Can't Prevent
At the end of the day, the client has a lot of information about the state of the game. It has information about what ores are underground. It has information about where other players might be hiding. It knows the contents of chests.
So people can always hack a client to make that sort of thing visible.
However, most people that do that would not use your mod, but would create a mod specifically to do that. So your main objective is to not make security holes in your mod that can be taken advantage of.
This philosophy (that all important logic be validated on the server) is good to apply to all client-server architectures, and most multi-player games are client-server.