If the first load method in PluginBroker :

$locator  = $this->getServiceLocator();
        // Pulling by alias
        if ($locator) {
            try {
                $instance = $locator->get($plugin);
            } catch (ServiceManagerException $e) {
                // not returning an instance is okay; there are other ways to 
                // retrieve the plugin later
                $instance = false;
            }

            if ($instance) {
                if ($this->getRegisterPluginsOnLoad()) {
                    $this->register($pluginName, $instance);
                }
                return $instance;
            }
        }

With these lines i can write in my controller :

public function indexAction()
    {
        $manager = $this->plugin('ViewManager'));
    }

I think is not the PluginBroker job, so to fix that i propose to verify alias configuration with :

$class = $this->getClassLoader()->load($plugin);
if (empty($class) && !class_exists($plugin)) {
            throw new Exception\RuntimeException('Unable to locate class associated with "' . $pluginName . '"');
        }

just before the loading by alias and keep the use of servicemanager to load alias and complete class name.