Phpactor Extensions

Over the past month or so I have been gradually migrating Phpactor to use Extensions.

This started because I wanted to add Language Server capabilities to Phpactor, but having two RPC mechanisms in the same application seemed overkill, so I decided to extract everything into extensions in order that all of the components could be easily reused and recombined (so that a phpactor-language-server standalone application could be created).

In addition I wanted the ability to add framework and tool specific functionality, which doesn't belong in the main distribution. This all pointed the way to having user extensions.

Installing Extensions

Writing an Extension

Extensions have a few key attributes:

  1. The extension package should have a package type of phpactor-extension and an extra attribute phpactor.extension_class which points to...
  2. The extension class which implements Phpactor\Container\Extension.

That's it. The extension class is just a DI container (similar to Pimple but with tags and parameters) with additional configuration (something like the Symfony Option Resolver).

Stupid Completor

DISCLAIMER: Phpactor is not currently not stable, and some packages have no tagged release at all.

Lets make a completion extension. This extension will accept some configuration: stupid_completor.items and it will return these items as suggestions every time it is invoked.

First of all we will need to require the phpactor/container package (this is the only strict requirement) and the phpactor/completion-extension (as we are building a completor) and ensure our composer file has the following attributes:

  1. A type of phpactor-extension
  2. An extra property with the FQN of the extension class.

It might look something like this:

{
    "name": "acme/stupid-completion-extension",
    "description": "Stupid Completion Support",
    "license": "MIT",
    "type": "phpactor-extension",
    "minimum-stability": "dev",
    "require": {
        "phpactor/container": "^1.0",
        "phpactor/completion-extension": "~0.1",
    },
    "autoload": {
        "psr-4": {
            "Acme\\Extension\\StupidCompletion\\": "lib/"
        }
    },
    "extra": {
        "phpactor.extension_class": "Acme\\Extension\\StupidCompletion\\StupidCompletionExtension"
    }
}

NOTE: that the completion extension has no release at time of writing so minimum-stability: dev is currently required.

We need to create a completor class to provide our stupid suggestions, let's put it in lib/Completion/StupidCompletion.php:

<?php

namespace Acme\Extension\StupidCompletion\Completion;

use Generator;
use Phpactor\Completion\Core\Completor;
use Phpactor\Completion\Core\Suggestion;

class StupidCompletion implements Completor
{
    private $suggestions;

    public function __construct(array $suggestions)
    {
        $this->suggestions = $suggestions;
    }

    public function complete(string $source, int $byteOffset): Generator
    {
        foreach ($this->suggestions as $suggestion) {
            yield Suggestion::create($suggestion);
        }
    }
}

Now we need the extension class, this will integrate our completor, this should be in lib/StupidCompletionExtension.php as with the above:

<?php

namespace Acme\Extension\StupidCompletion;

use Acme\Extension\StupidCompletion\Completion\StupidCompletion;
use Phpactor\Container\Container;
use Phpactor\Container\ContainerBuilder;
use Phpactor\Container\Extension;
use Phpactor\Extension\Completion\CompletionExtension;
use Phpactor\MapResolver\Resolver;

class StupidCompletionExtension implements Extension
{
    public const PARAM_ITEMS = 'stupid_completor.items';

    public function load(ContainerBuilder $container)
    {
        $container->register('stupid_completor.stupid_completor', function (Container $container) {
            return new StupidCompletion(
                $container->getParameter(self::PARAM_ITEMS)
            );
        }, [ CompletionExtension::TAG_COMPLETOR => []]);
    }

    public function configure(Resolver $schema)
    {
        $schema->setDefaults([
            self::PARAM_ITEMS => [
                'hello', 'goodbye'
            ]
        ]);
    }
}

Note that above:

  1. We add a tag to our completor from the CompletionExtension. Anything that is "public" is exposed as a public constant, including tags and services (TAG_* and SERVICE_*).
  2. We set some default configuration, when used with Phpactor this can be set in .phpactor.yml as stupid_completor.items.

Testing it Out

You could probably now push your extension to packagist, or add it as a path repository in Phpactor's extensions/extensions.json file (which is actually a composer.json file):

    "repositories": [
        {
            "type": "path",
            "url": "\/home\/daniel\/www\/phpactor\/stupid-completor-extension"
        }
    ]

Once this is done you are ready to install it with:

$ ~/.vim/plugged/phpactor/bin/phpactor extension:install acme/stupid-completion-extension

Note that Phpactor will load extensions based on the contents of the file extensions/extensions.php - if you experience issues you may want to disable the extension temporarily in this file.

Making a Standalone Application

Sometimes you might create an extension which can be used standalone. This is beneficial for user testing and if the extension can be useful without Phpactor.

Our standalone application will provide completion results over Phpactor's RPC protocol and will need the command line interface, so require the following:

$ composer require phpactor/completion-rpc-extension phpactor/console-extension

Create a standalone RPC application for stupid completion: just create the following file in bin/stupid-completion:

#!/usr/bin/env php
<?php

use Acme\Extension\StupidCompletion\StupidCompletionExtension;
use Phpactor\Container\PhpactorContainer;
use Phpactor\Extension\Completion\CompletionExtension;
use Phpactor\Extension\Console\ConsoleExtension;
use Phpactor\Extension\Logger\LoggingExtension;
use Phpactor\Extension\Rpc\RpcExtension;
use Phpactor\FilePathResolverExtension\FilePathResolverExtension;
use Symfony\Component\Console\Application;

require __DIR__ . '/../vendor/autoload.php';

$container = PhpactorContainer::fromExtensions([
    StupidCompletionExtension::class,
    CompletionExtension::class,
    ConsoleExtension::class,
    RpcExtension::class,
    LoggingExtension::class,
    FilePathResolverExtension::class,
], []);

$application = new Application();
$application->setCommandLoader(
    $container->get(ConsoleExtension::SERVICE_COMMAND_LOADER)
);
$application->run();

Note that:

  1. We instantiate a PhpactorContainer
  2. We manually added all the required extensions (the container will shout at you if any extensions were missing).
  3. We create a new Symfony Application and retrieve the command loader from the console extension.
  4. We run the application

Make it executable with chmod a+x bin/stupid-completion and now you have a stupid RPC completor!

$ echo '{"action": "complete", "parameters": {"source": "<?php ", "offset": 2}}' | ./bin/stupid rpc --pretty
{
    "version": "1.0.0",
    "action": "return",
    "parameters": {
        "value": {
            "suggestions": [
                {
                    "type": null,
                    "name": "hello",
                    "label": "hello",
                    "short_description": null,
                    "class_import": null,
                    "info": null
                },
                {
                    "type": null,
                    "name": "goodbye",
                    "label": "goodbye",
                    "short_description": null,
                    "class_import": null,
                    "info": null
                }
            ],
            "issues": []
        }
    }
}

Summary

Extensions should allow Phpactor to be extended in all sorts of ways, as well as providing a very fast way to create entirely new applications based on Phpactor functionality.

The above extension ommits tests for the completor and the extension itself. For a simple(ish) working example see the behat extension.

Rephpactor

TL;DR

Phpactor 1.0 will have no features at all, but it will provide a way to install extensions. All current Phpactor functionality will be extracted to extensions.

Background

One problem with Phpactor has always been that it has not been extensible - it is not possible to, for example, install a Behat extension, or a Phpspec or Symfony extension.

It is not that the infrastructure isn't there internally - it is and was based on the precedent set by Phpbench (which was in turn influenced by other things, notaby Behat, Symfony, Pimple, etc).

Phpbench could be easily included as a dependency of your project, this meant that it was easy to simply include the extension in your project as you would any other library.

Phpactor is a standalone project, you (generally) install it one place and use it everywhere. While you could include new dependencies on the project, it would not be a good idea because you will have conflicts when updating.

Scaling

Another problem has been that Phpactor has been aggregating functionality, and as time has gone on I wish that I could drop certain things, or introduce new domain-specific features.

Another long-standing problem has been lack of code fixers (prettifiers). While I have been tempted to write a Phpactor CS Fixer, it would only have been able to do the absolute minimum to fix the grossest formatting errors in generated code. So it makes far sense to make use of an existing tools php-cs-fixer and phpcs - but it makes not so much sense to bind them to Phpactor, as people will want to use one or the other (often depending on project requirements).

The Language Server

Recently I have been playing with a Phpactor Language Server Protocol (LSP) implementation, I have introduced this into the develop branch, it is generally works quite well. The biggest advantage is that it opens Phpactor up to other text editors with no additional effort, and it means ultimately not having to maintain a phpactor.vim plugin.

The disadvantage is that it's a long running process, and at the moment at least the original Phpactor is more stable.

Anyway - it leads to a problem where more code is added to the core which duplicates existing functionality and introduces more noise. It would be much better if the language server were optional.

Extensions

So this weekend I played with the idea of introducing an embedded composer. After checking out Beau Simensen's embedded composer. I managed to get a stripped down embedded composer working in a prototype project: rephpactor.

Rephpactor

Rephpactor (which will hopefully become Phpactor 1.0) will look something like this with no extensions installed:

Rephpactor

Usage:
  command [options] [arguments]

Options:
  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
  help               Displays help for a command
  list               Lists commands
 extension
  extension:install  Install extension
  extension:update   Update extensions
  extension:search   Search available extensions
  extension:list     List installed extensions

There is absolutely nothing there! It's amazing.

There is absolutely nothing there! It's amazing.

After initially installing you will be able to use the extension:install command to add packages from Packagist (only those with the phpactor-extension) type are permitted:

$ ./bin/rephpactor extension:install phpactor/language-server-extension

The installed extensions can then be listed:

$ ./bin/rephpactor extension:list
+--------------------------------------+-----------+--------------------------------------+
| Name                                 | Version   | Description                          |
+--------------------------------------+-----------+--------------------------------------+
| phpactor/language-server-extension   | 1.0.x-dev | LSP compatible language server       |
| phpactor/completion-extension        | 1.0.x-dev | Completion framework                 |
| phpactor/worse-reflection-extension  | 1.0.x-dev | Completors and other terrbile things |
+--------------------------------------+-----------+--------------------------------------+

Profit

This change, when it makes it to Phpactor, will make it possible to support more diverse domains. So for example, Symfony DI Completion, or Behat "feature to step jumping". Things get even more interesting at the language-server level.

It would be easy to create for example a PHPStan extension for the language server (and fulfil the LSP APIs for diagnostics) or a php-cs-fixer extension (and fulfil the LSP APIs for code formatting). It would even be possible to add completors based on existing tools (such as Psalm).

The most important thing is, that by removing pretty much everything from Phpactor by default, we can release a stable 1.0 version and there would be much rejoicing.

Feature Agnostic

As a foot note, Phpactor would also be agnostic to function. It would no longer need to do anything related to PHP code development, it essentially just provides a way to install extensions and bootstrap commands.

Three Years of Phpactor

The first commit in Phpactor (pronounced "factor") dates from almost three years ago:

commit 3677c5cb58a5b203fb658c8e2498e512cdef555a
Author: dantleech <dan.t.leech@gmail.com>
Date:   Thu Sep 24 14:08:35 2015 +0200

    Initial

I had no idea about how to create such an ambitious project in a domain in which I knew nothing. But I had been using VIM for around 7 years (?), VIM is a great text editor, but the tooling around refactoring and auto-completion for PHP was sub-optimal, and instead of waiting more years, I decided to write my own tool.

Actually almost all Phpactor development has happened in the past year-and-a-half. The above commit was the first of three attempts to create a code-completion and refactoring tool backend for editors such as VIM. This initial (3 commit) effort was to use an SQLite database to index the classes and functions in a project (an approach that was later dropped). Then followed another few commits four months later, then more 6 months after that. More serious development started in late 2016 but I struggled with the PhpParser, I then found out about the Microsoft Tolerant PHP Parser which was designed exactly for Phpactor's use case, I also decided to take a more pragmatic approach to the project.

The Phpactor logo

The Phpactor logo

What single thing would deliver the most value to me, what would provide the biggest return-on-investment?

Completion would have been nice, but the one thing that I wanted the most was a way to move classes - this has been an extremely painful thing to do in VIM, requiring not only moving files, but also updating all of the class namespaces, and all the references to those classes.

I decided to concentrate on this single feature instead of trying to solve the much more difficult problem of code completion. I would just do whatever was necessary to make class moving work, in a separate, fully decoupled, stand-alone library. It wouldn't matter if the code was sub-optimal as it wouldn't contaminate other areas of the application. With this in mind I restarted the project:

commit 07a8bbb442966854bc6029e7e8490b151366e69a
Author: dantleech <dan.t.leech@gmail.com>
Date:   Mon Jun 19 14:47:32 2017 +0100

    Restarting the project

From this point on Phpactor has just continued to grow and it slowly aggregated more and more functionality spread out over different repositories. The class moving library has basically remained the same since it's creation.

Parameter Completion

Parameter Completion

The origins of Phpactor can be found in some humble VIM plugins I wrote, one which determines the namespace of the current class file and another which generates a PHPUnit test case for the current class. Both of these plugins made use of the composer autoloader to determine class locations. This non-standard use of the composer autoloader is what powers Phpactor's source-location abilities, making slow indexing processes and caching largely unnecessary.

The current version of Phpactor is something of an epic project. It has many libraries (for example worse reflection, code-transform, code-builder, docblock, class-mover, completion, path-finder, and more). All of the libraries are untagged and unstable, none of them are intended to be consumed by other projects at this point in time and most of them fall short of being outstanding, providing only what is good-enough for the Phpactor project. They are, however decoupled from the main Phpactor application.

Components

Some of Phpactors Components

An Opportunity to Experiment

One of the advantages of personal projects is that you have freedom to experiment with new ways of doing things, while at work this can either be risky, or inappropriate.

In writing this project I wanted to try some of the DDD concepts I discovered after reading Vaughn Vernons Implementing Domain-Driven Design. Implementing a new paradigm is always going to be a trail-and-error experience, and I would do some things differently next time. All the libraries in Phpactor have a directory structure similar to the following:

lib/
    Core/
        ...
    Adapter/
        ...

With all the "clean" uncoupled code in Core (I was going to call this Domain, but didn't want to presume that I was doing DDD) and the adapter which implement the interfaces in Core and provide a coupling to another library. In an ideal world these adapters would be in separate packages, but the value wouldn't outweigh the effort in this case (or at least at this moment in time). I also implemented many Value Objects (VOs).

There is an amount of VO duplication between packages, notably for things such as SourceCode and ClassName objects. It might make sense in the future to extract some of the VO objects to a separate packages, but it's difficult to determine if the meaning is exactly the same (e.g. a ClassName VO in a library which infers class names from filenames has different requirements than a ClassName in the reflection library), however SourceCode is implemented three or four times as basically the same class.

Extract Method

Extract Method

I also learnt to avoid expose value objects at package boundaries as arguments. Exposing them meant that code outside the package would know more about the package than it ought to have done (for example, it imported the class name, or used a specific method of the VO) and therefore increased the difficulty and risk of refactoring the package.

For example:

use Phpactor\\ClassToFile\\Core\\ClassName as ClassToFileClassName;
use Phpactor\\WorseReflection\\Core\\ClassName as ReflectionClassName;

$file = $classToFile->classToFile(ClassToFileClassName::fromString('Foobar\\Barfoo'));
$reflection = $reflector->reflectClass(ReflectionClassName::fromString('Foobar\\Barfoo'));

and without the VO:

$file = $classToFile->classToFile('Foobar\\Barfoo');
$reflection = $reflector->reflectClass('Foobar\\Barfoo');

Internally the package could still use the VO but this detail is hidden from the outside.

Wheel Reinventing

The wheel has been reinvented a few times, notably in the case of WorseReflection (WR) - the backbone of Phpactor. It provides broadly the same functionality as, and was influenced by, BetterReflection (BR) with the addition of type and value flow (required for completion). The justification here is that it would have been impossible to merge the type-flow code in WR into BR, because it was so bad and experimental. But whilst being experimental it was providing actual value to Phpactor. In addition, BR had some performance problems (at the time) which made it specifically unsuitable for real-time completion.

On one hand it is a shame that I didn't contribute to BetterReflection, but on the other I don't think Phpactor would have been built if I did. WR is the core domain, and as such it is subservient to the needs of the project and needs to be owned by it.

Another example of wheel-reinventing is the docblock parser. There is already the PHPDoc DocBlock and the great PHPStan PHPDoc Parser. The first project depended on the nikic/php-parser for type resolution (which is arguably not a requirement for a parser). The PHPStan parser was functionally perfect, and I happily tried to replace the Phpactor parser - but unfortunately it was 10x slower than dumb regex parsing, so the otherwise inferior Phpactor package is still relevant. It's the difference between a 0.25s completion time on a PHPUnit test case, and a 2.5s one.

Finally there is Phpactor's RPC protocol used to talk to the editor. At the time I was vaguely aware of Langauge Server Protocol (LSP) but didn't look more into it as it is for a language server. Phpactor is not a server, it's invoked as a command. In hindsight the RPC protocol of Phpactor can fit inside the LSP and Phpactor could optionally be made into a server (although running as a short-lived process is better in terms of stability) (see pull request). LSP support would allow Phpactor to be used transparently by many more editors.

Implement Contract

Import class and Implement Contract

Return on Investment

Phpactor has taken up a huge amount of my spare time over the past year-and-a-half. I enjoy coding and look forward to spending a Saturday morning crafting a new feature in Phpactor, but often morning becomes mid-afternoon, and sometimes intrudes into Sunday.

Personally Phpactor is now an indispensable tool that I use at work every day, and I am most motivated to work on it when I am presented with a particular challenge in my job.

But I do ask myself if it is worth the time given that there are other projects which have grown in Phpactors lifetime (e.g. Php Language Server and Padawan) but these two libraries are mostly (exclusively?) concerned with code-completion, I don't think there is anything freely available which competes directly with Phpactor.

But still - is it worth me investing all this time when I could be working on other projects? (like my other side-project, Phpbench, which has seen little attention since I started Phpactor) -- or -- doing things other than programming?

This is a question I ask myself sometimes, and to be honest, all things considered, it probably isn't worth it. But I am happy that Phpactor turns VIM into viable modern IDE for PHP and it can now handle finding method references for example, and provide some relief to users of other editors working with VIM users:

I was slightly surprised to notice when I paired with a developer using PHPStorm we both look for references to a particular method, both PHPStorm and Phpactor returned the same methods and, when finding class references, Phpactor actually seemed to have out-performed PHPStorm. I do not assert that Phpactor is as accurate or comprehensive as PHPStorm (because it is not), but it does a pretty good job.

Class References

Class References

Finally, there is more interaction with other people in the Phpactor project than on my other projects, although it only has ~280 stars on Github (compared to, for example, Phpbench's ~780) there are many more people contributing and raising issues and creating third-party integrations (such as a plugin for emacs and integrations with completion manages such as ncm and deoplete). Having this feedback is encouraging,

Phpactor is by no means perfect - it will not find all your references, it will not complete everthing that PHPStorm would, some of the refactorings will, sometimes leave you with incorrect code. But for the most part it works really well.

It may be that one day Phpactor will be displaced by an even better solution, which would be fine. But I hope it will continue to grow and that some of the technical debt can be repaid and that one day some of the libraries will be stable and even more useful refactorings and features will be developed.