Rephpactor

TL;DR

Phpactor 1.0 will have no features at all, but it will provide a way to install extensions. All current Phpactor functionality will be extracted to extensions.

Background

One problem with Phpactor has always been that it has not been extensible - it is not possible to, for example, install a Behat extension, or a Phpspec or Symfony extension.

It is not that the infrastructure isn't there internally - it is and was based on the precedent set by Phpbench (which was in turn influenced by other things, notaby Behat, Symfony, Pimple, etc).

Phpbench could be easily included as a dependency of your project, this meant that it was easy to simply include the extension in your project as you would any other library.

Phpactor is a standalone project, you (generally) install it one place and use it everywhere. While you could include new dependencies on the project, it would not be a good idea because you will have conflicts when updating.

Scaling

Another problem has been that Phpactor has been aggregating functionality, and as time has gone on I wish that I could drop certain things, or introduce new domain-specific features.

Another long-standing problem has been lack of code fixers (prettifiers). While I have been tempted to write a Phpactor CS Fixer, it would only have been able to do the absolute minimum to fix the grossest formatting errors in generated code. So it makes far sense to make use of an existing tools php-cs-fixer and phpcs - but it makes not so much sense to bind them to Phpactor, as people will want to use one or the other (often depending on project requirements).

The Language Server

Recently I have been playing with a Phpactor Language Server Protocol (LSP) implementation, I have introduced this into the develop branch, it is generally works quite well. The biggest advantage is that it opens Phpactor up to other text editors with no additional effort, and it means ultimately not having to maintain a phpactor.vim plugin.

The disadvantage is that it's a long running process, and at the moment at least the original Phpactor is more stable.

Anyway - it leads to a problem where more code is added to the core which duplicates existing functionality and introduces more noise. It would be much better if the language server were optional.

Extensions

So this weekend I played with the idea of introducing an embedded composer. After checking out Beau Simensen's embedded composer. I managed to get a stripped down embedded composer working in a prototype project: rephpactor.

Rephpactor

Rephpactor (which will hopefully become Phpactor 1.0) will look something like this with no extensions installed:

Rephpactor

Usage:
  command [options] [arguments]

Options:
  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
  help               Displays help for a command
  list               Lists commands
 extension
  extension:install  Install extension
  extension:update   Update extensions
  extension:search   Search available extensions
  extension:list     List installed extensions

There is absolutely nothing there! It's amazing.

There is absolutely nothing there! It's amazing.

After initially installing you will be able to use the extension:install command to add packages from Packagist (only those with the phpactor-extension) type are permitted:

$ ./bin/rephpactor extension:install phpactor/language-server-extension

The installed extensions can then be listed:

$ ./bin/rephpactor extension:list
+--------------------------------------+-----------+--------------------------------------+
| Name                                 | Version   | Description                          |
+--------------------------------------+-----------+--------------------------------------+
| phpactor/language-server-extension   | 1.0.x-dev | LSP compatible language server       |
| phpactor/completion-extension        | 1.0.x-dev | Completion framework                 |
| phpactor/worse-reflection-extension  | 1.0.x-dev | Completors and other terrbile things |
+--------------------------------------+-----------+--------------------------------------+

Profit

This change, when it makes it to Phpactor, will make it possible to support more diverse domains. So for example, Symfony DI Completion, or Behat "feature to step jumping". Things get even more interesting at the language-server level.

It would be easy to create for example a PHPStan extension for the language server (and fulfil the LSP APIs for diagnostics) or a php-cs-fixer extension (and fulfil the LSP APIs for code formatting). It would even be possible to add completors based on existing tools (such as Psalm).

The most important thing is, that by removing pretty much everything from Phpactor by default, we can release a stable 1.0 version and there would be much rejoicing.

Feature Agnostic

As a foot note, Phpactor would also be agnostic to function. It would no longer need to do anything related to PHP code development, it essentially just provides a way to install extensions and bootstrap commands.

Three Years of Phpactor

The first commit in Phpactor (pronounced "factor") dates from almost three years ago:

commit 3677c5cb58a5b203fb658c8e2498e512cdef555a
Author: dantleech <dan.t.leech@gmail.com>
Date:   Thu Sep 24 14:08:35 2015 +0200

    Initial

I had no idea about how to create such an ambitious project in a domain in which I knew nothing. But I had been using VIM for around 7 years (?), VIM is a great text editor, but the tooling around refactoring and auto-completion for PHP was sub-optimal, and instead of waiting more years, I decided to write my own tool.

Actually almost all Phpactor development has happened in the past year-and-a-half. The above commit was the first of three attempts to create a code-completion and refactoring tool backend for editors such as VIM. This initial (3 commit) effort was to use an SQLite database to index the classes and functions in a project (an approach that was later dropped). Then followed another few commits four months later, then more 6 months after that. More serious development started in late 2016 but I struggled with the PhpParser, I then found out about the Microsoft Tolerant PHP Parser which was designed exactly for Phpactor's use case, I also decided to take a more pragmatic approach to the project.

The Phpactor logo

The Phpactor logo

What single thing would deliver the most value to me, what would provide the biggest return-on-investment?

Completion would have been nice, but the one thing that I wanted the most was a way to move classes - this has been an extremely painful thing to do in VIM, requiring not only moving files, but also updating all of the class namespaces, and all the references to those classes.

I decided to concentrate on this single feature instead of trying to solve the much more difficult problem of code completion. I would just do whatever was necessary to make class moving work, in a separate, fully decoupled, stand-alone library. It wouldn't matter if the code was sub-optimal as it wouldn't contaminate other areas of the application. With this in mind I restarted the project:

commit 07a8bbb442966854bc6029e7e8490b151366e69a
Author: dantleech <dan.t.leech@gmail.com>
Date:   Mon Jun 19 14:47:32 2017 +0100

    Restarting the project

From this point on Phpactor has just continued to grow and it slowly aggregated more and more functionality spread out over different repositories. The class moving library has basically remained the same since it's creation.

Parameter Completion

Parameter Completion

The origins of Phpactor can be found in some humble VIM plugins I wrote, one which determines the namespace of the current class file and another which generates a PHPUnit test case for the current class. Both of these plugins made use of the composer autoloader to determine class locations. This non-standard use of the composer autoloader is what powers Phpactor's source-location abilities, making slow indexing processes and caching largely unnecessary.

The current version of Phpactor is something of an epic project. It has many libraries (for example worse reflection, code-transform, code-builder, docblock, class-mover, completion, path-finder, and more). All of the libraries are untagged and unstable, none of them are intended to be consumed by other projects at this point in time and most of them fall short of being outstanding, providing only what is good-enough for the Phpactor project. They are, however decoupled from the main Phpactor application.

Components

Some of Phpactors Components

An Opportunity to Experiment

One of the advantages of personal projects is that you have freedom to experiment with new ways of doing things, while at work this can either be risky, or inappropriate.

In writing this project I wanted to try some of the DDD concepts I discovered after reading Vaughn Vernons Implementing Domain-Driven Design. Implementing a new paradigm is always going to be a trail-and-error experience, and I would do some things differently next time. All the libraries in Phpactor have a directory structure similar to the following:

lib/
    Core/
        ...
    Adapter/
        ...

With all the "clean" uncoupled code in Core (I was going to call this Domain, but didn't want to presume that I was doing DDD) and the adapter which implement the interfaces in Core and provide a coupling to another library. In an ideal world these adapters would be in separate packages, but the value wouldn't outweigh the effort in this case (or at least at this moment in time). I also implemented many Value Objects (VOs).

There is an amount of VO duplication between packages, notably for things such as SourceCode and ClassName objects. It might make sense in the future to extract some of the VO objects to a separate packages, but it's difficult to determine if the meaning is exactly the same (e.g. a ClassName VO in a library which infers class names from filenames has different requirements than a ClassName in the reflection library), however SourceCode is implemented three or four times as basically the same class.

Extract Method

Extract Method

I also learnt to avoid expose value objects at package boundaries as arguments. Exposing them meant that code outside the package would know more about the package than it ought to have done (for example, it imported the class name, or used a specific method of the VO) and therefore increased the difficulty and risk of refactoring the package.

For example:

use Phpactor\\ClassToFile\\Core\\ClassName as ClassToFileClassName;
use Phpactor\\WorseReflection\\Core\\ClassName as ReflectionClassName;

$file = $classToFile->classToFile(ClassToFileClassName::fromString('Foobar\\Barfoo'));
$reflection = $reflector->reflectClass(ReflectionClassName::fromString('Foobar\\Barfoo'));

and without the VO:

$file = $classToFile->classToFile('Foobar\\Barfoo');
$reflection = $reflector->reflectClass('Foobar\\Barfoo');

Internally the package could still use the VO but this detail is hidden from the outside.

Wheel Reinventing

The wheel has been reinvented a few times, notably in the case of WorseReflection (WR) - the backbone of Phpactor. It provides broadly the same functionality as, and was influenced by, BetterReflection (BR) with the addition of type and value flow (required for completion). The justification here is that it would have been impossible to merge the type-flow code in WR into BR, because it was so bad and experimental. But whilst being experimental it was providing actual value to Phpactor. In addition, BR had some performance problems (at the time) which made it specifically unsuitable for real-time completion.

On one hand it is a shame that I didn't contribute to BetterReflection, but on the other I don't think Phpactor would have been built if I did. WR is the core domain, and as such it is subservient to the needs of the project and needs to be owned by it.

Another example of wheel-reinventing is the docblock parser. There is already the PHPDoc DocBlock and the great PHPStan PHPDoc Parser. The first project depended on the nikic/php-parser for type resolution (which is arguably not a requirement for a parser). The PHPStan parser was functionally perfect, and I happily tried to replace the Phpactor parser - but unfortunately it was 10x slower than dumb regex parsing, so the otherwise inferior Phpactor package is still relevant. It's the difference between a 0.25s completion time on a PHPUnit test case, and a 2.5s one.

Finally there is Phpactor's RPC protocol used to talk to the editor. At the time I was vaguely aware of Langauge Server Protocol (LSP) but didn't look more into it as it is for a language server. Phpactor is not a server, it's invoked as a command. In hindsight the RPC protocol of Phpactor can fit inside the LSP and Phpactor could optionally be made into a server (although running as a short-lived process is better in terms of stability) (see pull request). LSP support would allow Phpactor to be used transparently by many more editors.

Implement Contract

Import class and Implement Contract

Return on Investment

Phpactor has taken up a huge amount of my spare time over the past year-and-a-half. I enjoy coding and look forward to spending a Saturday morning crafting a new feature in Phpactor, but often morning becomes mid-afternoon, and sometimes intrudes into Sunday.

Personally Phpactor is now an indispensable tool that I use at work every day, and I am most motivated to work on it when I am presented with a particular challenge in my job.

But I do ask myself if it is worth the time given that there are other projects which have grown in Phpactors lifetime (e.g. Php Language Server and Padawan) but these two libraries are mostly (exclusively?) concerned with code-completion, I don't think there is anything freely available which competes directly with Phpactor.

But still - is it worth me investing all this time when I could be working on other projects? (like my other side-project, Phpbench, which has seen little attention since I started Phpactor) -- or -- doing things other than programming?

This is a question I ask myself sometimes, and to be honest, all things considered, it probably isn't worth it. But I am happy that Phpactor turns VIM into viable modern IDE for PHP and it can now handle finding method references for example, and provide some relief to users of other editors working with VIM users:

I was slightly surprised to notice when I paired with a developer using PHPStorm we both look for references to a particular method, both PHPStorm and Phpactor returned the same methods and, when finding class references, Phpactor actually seemed to have out-performed PHPStorm. I do not assert that Phpactor is as accurate or comprehensive as PHPStorm (because it is not), but it does a pretty good job.

Class References

Class References

Finally, there is more interaction with other people in the Phpactor project than on my other projects, although it only has ~280 stars on Github (compared to, for example, Phpbench's ~780) there are many more people contributing and raising issues and creating third-party integrations (such as a plugin for emacs and integrations with completion manages such as ncm and deoplete). Having this feedback is encouraging,

Phpactor is by no means perfect - it will not find all your references, it will not complete everthing that PHPStorm would, some of the refactorings will, sometimes leave you with incorrect code. But for the most part it works really well.

It may be that one day Phpactor will be displaced by an even better solution, which would be fine. But I hope it will continue to grow and that some of the technical debt can be repaid and that one day some of the libraries will be stable and even more useful refactorings and features will be developed.

Trelleborg to Berlin

Distance: 22.85km Time: 1h16m

I am now sitting on my cheap Ikea office chair typing this blog post on my desk in my apartment in Berlin.

This morning I woke first naturally at 4:30am, and then was woken again by the alarm clock on my phone at 5am. I raised myself in my sleeping bag and unzipped the panel which opens onto the "porch" of my tent. I had no muesli today, but had good bread, and I pasted it with jam and had my coffee and read the one day old news on my phone (there was no Wifi at the campsite).

I was advised to be at the ferry port at 6:30am for the 7:30am sailing, but I wanted to be there at 6am to give myself some breathing space. I was only 2km from the ferry port, and I left the campsite at 5:50am, and arrived at around 6:10am.

I was told that I could wait in "here" in the lounge or outside and at 7am somebody would "open the gate". I said I would wait outside assuming there was another lounge, but upon investigation it was just a fence with a gate, and I returned to the lounge and waited with some other passengers.

A man in a high-vis jacket came in and started ushering people towards the door, he consulted the girl at the check-in and she pointed at me and he beckoned me to come too and I was allowed to put my bike on the bus which drove onto the ferry.

I wasn't 100% sure that I was heading to the correct ferry, as we were driving towards a ferry with "Sassnitz" written in large letters on its hull, but we passed it and drove up a ramp towards a ferry with no discernible writing on the side, but given there were only two sailings I was happy to assume this was the Rostock ferry.

The bus driver waved me towards the end of the ferry and I free-wheeled my bike in that direction and another ferry operative made arm gestures towards the other bikes and I made the same arm gesture back and nodded. There were several bikes tied to the bars at the front of the car deck. There were no more "ties" however, so I just parked my bike with the kick-stand and made my way to the stairs.

After being lost for several minutes in the corridors and car levels I found my way to some of the passenger lounges. People were already there sleeping on the floor, some had brought duvets up from their cars. I suppose some people must have had a very early start or may not have even slept at all. One of the rooms smelt very badly of sewage, which is a shame as it was the room with the most comfortable chairs.

People were by this time helping themselves to breakfast, I got myself a coffee and a chocolate muffin and sat down and worked on my software project for a while, there was free Wifi, even if it was very slow indeed. I had a missed (whats-app) call from my mother which I only noticed when I connected to the Wifi, and I rang her. She was still sleeping, I didn't consider that it was 6am in England.

I did some reading on the ferry, and dozed a bit and worked a bit. We got into Rostock at around 13:30, making it a six hour crossing.

I had checked the train times on the internet before leaving the ferry and the next one was at 14:30, I had an hour to get to the train station, buy a ticket and get on the train. There was no absolute pressure, as the next was at 16:30, but I'd rather avoid the wait, so I made haste on the 10km ride to Rostock HBF (train station).

Sailing into Rostock

Sailing into Rostock

By the time I had found the ticket office and purchased my tickets (~€35 for me, ~€5 for the bike) I still had 10 minutes to spare. To access the tracks there were stairs, or a lift. There was a queue for the lift, a queue of four bicycles, and it seemed to be one biycle at a time. I queued for a few minutes, and then lost patience and decided to carry my bike down the stairs.

When I lifted the bike it seemed much lighter then I remember. This may have been because the water bottles were almost empty, or perhaps somehow I have become stronger in my arms despite doing all my exercise with my legs, or perhaps I have left something important and heavy behind - but it was easy work taking it down and then up the stairs to platform eight where I immediately saw the bicycle carriage.

There were already several bicycles in the carriage, and I added mine to the mess. The train was very busy. I took my baggage off and found the next available seat in the adjacent carriage, where I could just about see my bike.

I didn't really secure it, as doing so would have made the bikes under mine somewhat inaccessible and was made nervous when I saw a toddler walking up and down between the bicycles, and playing with my pedals - the little shit derailed my chain - but I was more worried about the train going around a corner and the bicycle falling on him. So I went up to check it was stable, and it was - and the father of the child said "It doesn't move, it's perfect".

I didn't do much at all in the 3 hour journey in the packed train, more and more passengers and bicycles kept piling in as we went. As we approached Berlin there were four different stations, the second was the HBF (central train station), but the first I didn't recognize. Five minutes before we arrived I searched on my phone I realised the first would be closer to my flat, so I hastily collected my four panniers (hanging them around my neck and arms) and shuffled my way towards the bicycle carriage, displacing people along the way. There were four bikes piled on top of mine now (which had made it's way to the back). Fortunately there were people on hand to help me out and as I wheeled my bike onto the platform the toddler-father handed me the last of my panniers and smiled "good luck" and I smiled back "Danke".

Berlin Gesundbrunner

Berlin Gesundbrunner

As I exited the station I recognized the place, although I had only been there once and that was on a run. In short time I was at my building. I almost forgot about my keys, I had removed them from my key fob and put them in my handle-bar basket, they were fortunately still there, although they had rusted somewhat.

I wheeled the bicycle into the corridor and hauled the bags off and heaved them up the 4 flights of stairs, then went back down for the bike, which was just in front of the mail boxes. I looked at my mail box and it was packed absolutely full of junk mail.

Junk Mail

My Junk Mail

It's probably like this in every city, but the local Netto supermarket sends a full fucking catalog to every single mailbox every single week, and then there are miscellaneous other tree crimes, but the catalog is the worst. I had been meaning to put a "No Junk Mail Please" sign on the box, and decided to do it now while I had a pen and paper.

After bringing the bike up the stairs I opened the door. I was half expecting an infestation of fruit-flies. I had mismanaged my garbage and the little shits had started hatching and I was trying to get rid of them when I left. My last tactic was to leave several bowls of red-wine around the flat. I'm not sure if that worked (the red wine had dried out and there were numerous dark spots, which may have been the flies) but in any case they were all gone (save one which I found in my shower).

Bike in Room

New Exhibition Piece

After depositing my bike, I went to the supermarket and got supplies - bread, shower gel (I used my last drops the night before), honey, soya-milk, nuts, other stuff, and a 6-pack of Störtebecker beer (with multiple varieties). I already had a four-cheese pizza in my freezer.

Being back in the flat is an anti-climax. I was looking forward to being home, but at the same time I wasn't. Being on the road there is always an aim to each day - and a larger aim (e.g. get to Trondheim, return to Berlin) and everything between the start of the day and the end of the day is an adventure towards that aim.

Being on the road, you kind of forecast your future each day. You decide that in 2 weeks you will be in this geographic location, and today you might be somewhere in this area, and you might possibly stay at this place. In between A and B anything can happen. The only thing that is almost guaranteed is that you will have exhausted yourself by the days end and feel good about it, even if, after the seventh hour you felt dreadful and every kilometer seems like the previous ten. After an arduous day nothing is more satisfying than well-deserved sleep.