West Nile virus and dengue virus are mosquito-borne flaviviruses that cause a large number of human infections each year. No vaccines or chemotherapeutics are currently available. These viruses encode a serine protease that is essential for polyprotein processing, a required step in the viral replication cycle. In this study, a high-throughput screening assay for the West Nile virus protease was employed to screen approximately 32,000 small-molecule compounds for identification of inhibitors. Lead inhibitor compounds with three distinct core chemical structures (1 to 3) were identified. In a secondary screening of selected compounds, two compounds, belonging to the 8-hydroxyquinoline family (compounds A and B) and containing core structure 1, were identified as potent inhibitors of the West Nile virus protease, with K(i) values of 3.2 +/- 0.3 microM and 3.4 +/- 0.6 microM, respectively. These compounds inhibited the dengue virus type 2 protease with K(i) values of 28.6 +/- 5.1 microM and 30.2 +/- 8.6 microM, respectively, showing some selectivity in the inhibition of these viral proteases. However, the compounds show no inhibition of cellular serine proteases, trypsin, or factor Xa. Kinetic analysis and molecular docking of compound B onto the known crystal structure of the West Nile virus protease indicate that the inhibitor binds in the substrate-binding cleft. Furthermore, compound B was capable of inhibiting West Nile virus RNA replication in cultured Vero cells (50% effective concentration, 1.4 +/- 0.4 microM; selectivity index, 100), presumably by inhibition of polyprotein processing.