A PSP ISO is a bitwise copy of the Universal Media Disc (UMD). It contains the file system (ISO 9660), the executable files (EBOOT.BIN), and asset files. The PSP executable format is typically an encrypted ELF file wrapped in a PBP container for digital distribution, or raw binary data on a UMD.
You cannot install custom PKG files on a standard, unmodified PS3. Sony only allows signed PKGs from the PlayStation Store. To install a converted PKG, your PS3 must be running: psp iso to pkg converter ps3