Reverse engineering in software engineering involves taking apart computer programs to understand how they work without having access to the original source code. It is done to retrieve lost source code, study how a program performs operations, improve performance, fix bugs, or adapt a program for a new processor. Tools like hexadecimal dumpers, disassemblers, and debuggers are used to reverse engineer programs by displaying the binary machine code in a readable format and avoiding disassembling data portions. There are different types of reverse engineering, including hardware and 3D imaging versions, but it risks legal issues if not done carefully to avoid claims of intellectual property theft.