We propose a 3D object detection and pose estimation method for automated driving using stereo images. In contrast to existing stereo-based approaches, we focus not only on cars, but on all types of road users and can ensure real-time capability through GPU implementation of the entire processing chain. These are essential conditions to exploit an algorithm for highly automated driving. Semantic information is provided by a deep convolutional neural network and used together with disparity and geometric constraints to recover accurate 3D bounding boxes. Experiments on the challenging KITTI 3D object detection benchmark show results that are within the range of the best image-based algorithms, while the runtime is only about a fifth. This makes our algorithm the first real-time image-based approach on KITTI.