Much medical data today remain inaccessible thus limiting their impact on patient care. Images and illustrations, scientific articles, and free-text reports do not allow for easy extraction and re-use of the knowledge they contain. They lack the structure and metadata necessary for automated processing and annotation. The resources required to collect and annotate manually are not sufficient to produce enough comprehensive benchmark datasets to bootstrap specialty research. We discuss neural network-based approaches to the problem of extraction of medical information from clinical images and unstructured text sources.