Document new source finding behaviour

This should cover the current state on master, as previously
discussed / implemented across python#9742, python#9683, python#9632, python#9616, python#9614, etc.

This will need to be changed if we can make `--namespace-packages` the
default (python#9636).

I haven't documented some of the finer points of the changes, since it
felt like an inappropriate level of detail (e.g. using absolute paths
when crawling, how directories with invalid package names affect
crawling, etc)

Author: hauntsaninja

docs/source/command_line.rst

@@ -110,10 +110,17 @@ imports.
     passed on the command line, the ``MYPYPATH`` environment variable,
     and the :confval:`mypy_path` config option.
 
-    Note that this only affects import discovery -- for modules and
-    packages explicitly passed on the command line, mypy still
-    searches for ``__init__.py[i]`` files in order to determine the
-    fully-qualified module/package name.
+    This flag also affects how mypy determines fully qualified module names for
+    files, modules and packages explicitly passed on the command line. See
+    :ref:`Mapping file paths to modules <mapping-paths-to-modules>` for details.
+
+.. option:: --explicit-package-bases
+
+    This flag tells mypy to that the current directory, the ``MYPYPATH``
+    environment variable, and the :confval:`mypy_path` config option as the
+    directories in which top-level packages are located. This option is only
+    useful in conjunction with :option:`--namespace-packages`. See :ref:`Mapping
+    file paths to modules <mapping-paths-to-modules>` for details.
 
 .. option:: --ignore-missing-imports
 

docs/source/running_mypy.rst

@@ -24,8 +24,11 @@ actual way mypy type checks your code, see our
 Specifying code to be checked
 *****************************
 
-Mypy lets you specify what files it should type check in several
-different ways.
+Mypy lets you specify what files it should type check in several different ways.
+
+Note that if you use namespace packages (in particular, packages without
+``__init__.py``), you'll need to specify :option:`--namespace-packages <mypy
+--namespace-packages>`.
 
 1.  First, you can pass in paths to Python files and directories you
     want to type check. For example::
@@ -336,58 +339,74 @@ while this option can be quite powerful, it can also cause many
 hard-to-debug errors.
 
 
-
 .. _mapping-paths-to-modules:
 
 Mapping file paths to modules
 *****************************
 
-One of the main ways you can tell mypy what files to type check
-is by providing mypy the paths to those files. For example::
+One of the main ways you can tell mypy what to type check
+is by providing mypy a list of paths. For example::
 
     $ mypy file_1.py foo/file_2.py file_3.pyi some/directory
 
 This section describes how exactly mypy maps the provided paths
 to modules to type check.
 
-- Files ending in ``.py`` (and stub files ending in ``.pyi``) are
-  checked as Python modules.
+- mypy will check all paths provided that correspond to files.
+
+- mypy will recursively discover and check all files ending in ``.py`` or
+  ``.pyi`` in directory paths provided.
 
-- Files not ending in ``.py`` or ``.pyi`` are assumed to be Python
-  scripts and checked as such.
+- For each file to be checked, mypy will attempt to associate the file (e.g.
+  ``project/foo/bar/baz.py``) with a fully qualified module name (e.g.
+  ``foo.bar.baz``). The directory the package is in (``project``) is then
+  add to mypy's module search paths.
 
-- Directories representing Python packages (i.e. containing a
-  ``__init__.py[i]`` file) are checked as Python packages; all
-  submodules and subpackages will be checked (subpackages must
-  themselves have a ``__init__.py[i]`` file).
+How mypy determines fully qualified module names depends on if the options
+:option:`--namespace-packages <mypy --namespace-packages>` and
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` are set.
 
-- Directories that don't represent Python packages (i.e. not directly
-  containing an ``__init__.py[i]`` file) are checked as follows:
+First, if :option:`--namespace-packages <mypy --namespace-packages>` is off,
+mypy will rely solely upon the presence of ``__init__.py[i]`` files to determine
+the fully qualified module name. That is, mypy will crawl up the directory tree
+for as long as it continues to find ``__init__.py`` (or ``__init__.pyi``) files.
 
-  - All ``*.py[i]`` files contained directly therein are checked as
-    toplevel Python modules;
+Second, if :option:`--namespace-packages <mypy --namespace-packages>` is on, but
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` is off, mypy
+will allow for the possibility that directories without ``__init__.py[i]`` are
+packages. Specifically, mypy will look at all parent directories of the file and
+use the location of the highest ``__init__.py[i]`` in the directory tree to
+determine the top-level package.
 
-  - All packages contained directly therein (i.e. immediate
-    subdirectories with an ``__init__.py[i]`` file) are checked as
-    toplevel Python packages.
+For example, say your directory tree consists solely of ``pkg/__init__.py`` and
+``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified module
+name, mypy will look at ``pkg/__init__.py`` and conclude that the associated
+module name is ``pkg.a.b.c.d.mod``.
 
-One more thing about checking modules and packages: if the directory
-*containing* a module or package specified on the command line has an
-``__init__.py[i]`` file, mypy assigns these an absolute module name by
-crawling up the path until no ``__init__.py[i]`` file is found.
+You'll notice that that method still relies on ``__init__.py``. If you can't put
+an ``__init__.py`` in your top-level package, but still wish to pass paths (as
+opposed to packages or modules using the ``-p`` or ``-m`` flags),
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` provides a
+solution.
 
-For example, suppose we run the command ``mypy foo/bar/baz.py`` where
-``foo/bar/__init__.py`` exists but ``foo/__init__.py`` does not.  Then
-the module name assumed is ``bar.baz`` and the directory ``foo`` is
-added to mypy's module search path.
+Third, with :option:`--explicit-package-bases <mypy --explicit-package-bases>`,
+mypy will locate the nearest parent directory that is a member of the
+``MYPYPATH`` environment variable, the :confval:`mypy_path` config or is the
+current working directory. mypy will then use the relative path to determine the
+fully qualified module name.
 
-On the other hand, if ``foo/bar/__init__.py`` did not exist, ``foo/bar``
-would be added to the module search path instead, and the module name
-assumed is just ``baz``.
+For example, say your directory tree consists solely of
+``src/namespace_pkg/mod.py``. If you run the command following command, mypy
+will correctly associate ``mod.py`` with ``namespace_pkg.mod``::
 
-If a script (a file not ending in ``.py[i]``) is processed, the module
-name assumed is ``__main__`` (matching the behavior of the
-Python interpreter), unless :option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.
+    $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases .
+
+If you pass a file not ending in ``.py[i]``, the module name assumed is
+``__main__`` (matching the behavior of the Python interpreter), unless
+:option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.
+
+Passing :option:`-v <mypy -v>` will show you the files and associated module
+names that mypy will check.
 
 
 .. _finding-imports:
@@ -407,7 +426,7 @@ This is computed from the following items:
   (a colon-separated list of directories).
 - The :confval:`mypy_path` config file option.
 - The directories containing the sources given on the command line
-  (see below).
+  (see :ref:`Mapping file paths to modules <mapping-paths-to-modules>`).
 - The installed packages marked as safe for type checking (see
   :ref:`PEP 561 support <installed-packages>`)
 - The relevant directories of the
@@ -418,11 +437,6 @@ This is computed from the following items:
     You cannot point to a :pep:`561` package via the ``MYPYPATH``, it must be
     installed (see :ref:`PEP 561 support <installed-packages>`)
 
-For sources given on the command line, the path is adjusted by crawling
-up from the given file or package to the nearest directory that does not
-contain an ``__init__.py`` or ``__init__.pyi`` file. If the given path
-is relative, it will only crawl as far as the current working directory.
-
 Second, mypy searches for stub files in addition to regular Python files
 and packages.
 The rules for searching for a module ``foo`` are as follows: