Key takeaways
|
Testing of uploading files can be tricky, but with the right tools and the knowledge of a few tricks, the process can be more efficient and a lot less difficult.
If you haven’t been exposed to vfsStream in the past, it allows interaction with a file that is stored in memory vs. one that is physically on the machine. This is nice since you don't have to delete files that are used for testing, which is more of a problem when a test fails and the teardown or other code doesn't run to remove the files used in testing. Additionally, it's faster since it is dealing with memory rather than the physical hard drive. Simply put, it's cleaner and faster.
This post will go over creating an endpoint (route) to upload a CSV file of users and testing that users in the CSV are displayed in the JSON response, as well as adding validation to ensure CSV files are the types of files being dealt with. While it is an overly realistic example, it should serve as a good launching point for implementing something similar in your projects.
For this overview, it is assumed you have Laravel (we use 5.3) installed. You won't need to actually hit this route, but it is recommend you use Laravel Valet or Homestead. This guide isn't going to go over getting those up and running. Here is a link to the repository where you can grab the code for the post.
Getting Started
Opening a console and navigating to the project root folder (where you can see the composer.json file listed alongside the .env file and others), if you run
composer require mikey179/vfsStream --dev
you will pull in vfsStream into the development section of the composer file and also pull in the needed files to leverage the package in testing. And with that, you should be ready to get testing.
Let’s Get to Some Code, Using TDD
To make this easy, simply make a file under the tests folder (from the project root again), and name it UploadCsvTest.php. Now, you need to open the file and put this inside it.
<?php
class UploadCsvTest extends TestCase
{
public function testSuccessfulUploadingCsv()
{
}
}
Start by getting the file upload working first, and then you can come back around and add the validation. The reason for doing it this way is so you can iteratively add functionality to working code. For example, if you can upload the file, that's working code, adding validation to it is enhancing it so that can be done later.
Now add the following code to "testSuccessfullUploadingCsv" function:
$this->json('POST', 'upload-new-users-csv', ['usersCsvFile' => 'someFile']);
$this->assertResponseOk();
From the console if you run this (later this will be referred to as PHPUnit)
phpunit --filter=UploadCsvTest
you should get, and expect, a failure - and it should look like this:
Expected status code 200, got 404.
Failed asserting that false is true.
Since a 404 means the route can't be found, let’s define that. Navigate to the routes/web.php file (The routes file moved here from the Http folder in previous version of Laravel) and add this to the bottom of the file:
Route::post('/upload-new-users-csv', function (\Illuminate\Http\Request $request) {
});
Now if you run PHPUnit, you will get a passing test. So, now it's time to access the file you should expect to get in the closure in the route file. So back in that file add the following line inside the closure:
$request->input('usersCsvFile')->openFile();
The Route should now look like this:
Route::post('/upload-new-users-csv', function (\Illuminate\Http\Request $request) {
$request->input('usersCsvFile')->openFile();
});
Now when you run PHPUnit, you are going to get another error, a 500. Figuring out what happened here is a little tricky. You can look at the content of the response by doing
dd($this->response->getContent());
in the test, but it’s important this is done before the asserting the response is OK, otherwise the test will fail before you get the output of the dd. The other way is to look at the log file located at "storage/logs/laravel.log". Either way you check, you should see a message similar to "Call to a member function openFile() on string", which is what you want because you sent a string through when you made the post in the test. Now it's time to get up to speed on vfs!
Getting Familiar with vfsStream
By no means is this comprehensive, but it's a start, and this is a very deep and powerful package. To make the test look a little nicer, begin by making the creation of the file in a function. So go back to the test file (UploadCsvTest.php) and make it look like this:
<?php
use Illuminate\Http\UploadedFile;
use org\bovigo\vfs\vfsStream;
class UploadCsvTest extends TestCase
{
public function testSuccessfulUploadingCsv()
{
$this->json('POST', 'upload-new-users-csv', ['usersCsvFile' => $this->createUploadFile()]);
$this->assertResponseOk();
}
protected function createUploadFile()
{
$vfs = vfsStream::setup(sys_get_temp_dir(), null, ['testFile' => 'someString']);
return new UploadedFile(
$vfs->url().'/testFile',
null,
'text/csv',
null,
null,
true
);
}
}
A lot has been introduced here; first the vfsStream::setup. This is creating a file in memory. The example uses the "sys_get_temp_dir()" just to be "more realistic", but anything could be used here, an example would be changing it to the string "root", and it will work just the same. The idea behind this is to allow you to create any directory structure to suit your needs.
Here it's not really important because PHP will automatically place uploaded files in the system temp directory; hence just using the "sys_get_temp_dir()", but it really doesn't matter in this case. The next parameter is setting the file permissions. In this case you don't need to get fancy with it so simply set it to null, allowing you to do what you want with the file. Finally there's the contents of directory ("/tmp" in the example), which is nothing more than an array of file name (key) and the value (a generic string for now, but it will need to be a CSV).
Lastly, you see the UploadedFile, which is an extension of Symfony's UploadedFile, but as a quick break down of the parameters in order (from the Symfony comment block):
First parameter: The full temporary path to the file
Second parameter: The original file name
Third parameter: The type of the file as provided by PHP; null defaults to application/octet-stream
Fourth parameter: The file size
Fifth parameter: The error constant of the upload (one of PHP's UPLOAD_ERR_XXX constants); null defaults to UPLOAD_ERR_OK
Sixth parameter: Whether the test mode is active
If you run PHPUnit, you should get a passing test! Now, time to make the file look a little more realistic. In the createUploadFile function, make it look like this:
protected function createUploadFile()
{
$vfs = vfsStream::setup(sys_get_temp_dir(), null, ['testFile.csv' => '']);
$uploadFile = new UploadedFile(
$vfs->url().'/testFile.csv',
null,
'text/csv',
null,
null,
true
);
collect([
['username', 'first name', 'last name'],
['jondoe', 'Jon', 'Doe'],
['janedoe', 'Jane', 'Doe']
])->each(function ($fields) use ($uploadFile) {
$uploadFile->openFile('a+')->fputcsv($fields);
});
return $uploadFile;
}
This leverages Laravel's collection helper to populate the file using some PHP functionality. The openFile() will get the underlying SPLFileObject and work with it directly, adding arrays to the file as if it were a line in a CSV. After running PHPUnit, you should still get green. Now, make sure you are getting what is expected, the JSON listing of the file in the response. To accomplish that, make the testSuccessfulUpload function look like this:
public function testSuccessfulUploadingCsv()
{
$this->json('POST', 'upload-new-users-csv', ['usersCsvFile' => $this->createUploadFile()])
->seeJson([
"username,\"first name\",\"last name\"\n","jondoe,Jon,Doe\n","janedoe,Jane,Doe\n"
]);
$this->assertResponseOk();
}
After running PHPunit, you will get red, as expected, so make that pass. Opening the web.php file, change the route closure to be this:
Route::post('/upload-new-users-csv', function (\Illuminate\Http\Request $request) {
$uploadedFile = $request->input('usersCsvFile')->openFile();
$returnArray = collect();
while (!$uploadedFile->eof()) {
$returnArray->push($uploadedFile->fgets());
}
return $returnArray;
});
All this does is read the CSV from the input (post) and places it into a collection so you can spit it out. Not a good real world example, but it's enough to get the test passing. So after running PHPUnit, you should get green.
Now, it’s time to create the test that going to make things break. There's a lot of work and preparation to this than one might think.
First, start with making the "createUploadFile" function more reusable so it can continue to pass the tests. The function should now look like this:
protected function createUploadFile(vfsStreamFile $file)
{
return new UploadedFile(
$file->url(),
null,
mime_content_type($file->url()),
null,
null,
true
);
}
Make sure to import (use) "org\bovigo\vfs\vfsStreamFile", or you will get error when you get to a state where green is expected. This will allow you to be able to make a file using vfsSream and it will know where (in memory) it's located and also determine the MIME type to make it.
Now you can create something that will make virtual files:
protected function createVirtualFile($filename, $extension)
{
return vfsStream::setup(sys_get_temp_dir(), null, [$filename.'.'.$extension => '']);
}
This is a simple function, not a lot going on. It simply makes a virtual file with the given name and extension. This helps to reduce duplication as you want to create a CSV and some other type later for testing.
Now, create a function that will utilize these functions to create a CSV, what you need now for the current test.
protected function createCsvUploadFile($fileName = 'testFile')
{
$virtualFile = $this->createVirtualFile($fileName, 'csv')->getChild($fileName.'.csv');
$fileResource = fopen($virtualFile->url(), 'a+');
collect([
['username', 'first name', 'last name'],
['jondoe', 'Jon', 'Doe'],
['janedoe', 'Jane', 'Doe']
])->each(function ($fields) use ($fileResource) {
fputcsv($fileResource, $fields);
});
fclose($fileResource);
return $this->createUploadFile($virtualFile);
}
This function is a little busier than the other two previously introduced. First, you’re making a CSV file of a given name, but then you utilize the "getChild" function, which is new. What this allows you to do is "pluck" the virtual file from the virtual system so you can work with it in a more direct manner. Previously, you didn't need to do this as you fed the file path directly to the UploadFile with this:
new UploadedFile(
$vfs->url().'/testFile.'.$extension, // This is the "manual" way of getting the child like we did above.
null,
text/csv', // This being hard carded in the old code makes us not be able to test different MIME types.
null,
null,
true
);
Now, back to the "createCsvUploadFile" function. After creating the file system and getting the CSV file out of it, you can use plain old PHP and Laravel to load the file up with your old data, making a CSV. Then pass it on over to the "createUploadFile" so you can get an upload file to work within your test ("testSuccessfulUploadingCsv") function.
Change what you need here to utilize that new "createCsvUploadFile" function, which is a pretty simple change:
// This line
$this->createUploadFile('.csv')
// Becomes
$this->createCsvUploadFile() // you could fill the file name in if you like. I added that parameter for future use if needed or wanted.
That’s it!
So now your test file should look like this:
<?php
use Illuminate\Http\UploadedFile;
use org\bovigo\vfs\vfsStream;
use org\bovigo\vfs\vfsStreamFile;
class UploadCsvTest extends TestCase
{
public function testSuccessfulUploadingCsv()
{
$this->json('POST', 'upload-new-users-csv', [
'usersCsvFile' => $this->createCsvUploadFile()
]);
$this->assertResponseOk();
$this->seeJson(["username,\"first name\",\"last name\"\n","jondoe,Jon,Doe\n","janedoe,Jane,Doe\n"]);
}
protected function createCsvUploadFile($fileName = 'testFile')
{
$virtualFile = $this->createVirtualFile($fileName, 'csv')->getChild($fileName.'.csv');
$fileResource = fopen($virtualFile->url(), 'a+');
collect([
['username', 'first name', 'last name'],
['jondoe', 'Jon', 'Doe'],
['janedoe', 'Jane', 'Doe']
])->each(function ($fields) use ($fileResource) {
fputcsv($fileResource, $fields);
});
fclose($fileResource);
return $this->createUploadFile($virtualFile);
}
protected function createUploadFile(vfsStreamFile $file)
{
return new UploadedFile(
$file->url(),
null,
mime_content_type($file->url()),
null,
null,
true
);
}
protected function createVirtualFile($filename, $extension)
{
return vfsStream::setup(sys_get_temp_dir(), null, [$filename.'.'.$extension => '']);
}
}
If you run the tests, you should get green (as expected).
Let’s get Back to Doing Some TDD
Now you can focus on trying to break the upload with a non CSV file. Since PHP can make image files relatively easily,that would be a good choice. Make a function that can make that kind of file and feed it into an upload file:
protected function createImageUploadFile($fileName = 'testFile', $extension = 'jpeg')
{
$virtualFile = $this->createVirtualFile($fileName, $extension)->getChild($fileName.'.'.$extension);
imagejpeg(imagecreate(500, 90), $virtualFile->url());
return $this->createUploadFile($virtualFile);
}
A pretty straight forward function here. Normally you may have just hardcoded the file type into the creation of the file, and in the getChild, but you want to try to trick this upload. This is done to test if there is an image - with a csv extension - to make sure it doesn't get fooled. So, after making the empty file, you will notice a jpeg (via the imagejpeg and imagecreate functions) in the file and then hand it over to the creation of the "createUploadFile" function, and back to its caller it goes. So, now all that's left to get some errors going is making the new function:
public function testOnlyCsvsCanBeUploadedRegardlessOfExtension()
{
$this->json('POST', 'upload-new-users-csv', [
'usersCsvFile' => $this->createImageUploadFile('testFile', 'csv')
]);
$this->assertResponseStatus(422);
}
This example is trying to upload that tricky file, a image file with a csv extension and you would expect a 422 (Unprocessable Entity). If you run phpunit, you should get an error, something like "Expected status code 422, got 500." This is perfect; it failed exactly as expected.
Now it's time to leverage Laravel's Form Requests. You can use Artisan to do this easily. Back in the console, if you run:
php artisan make:request UploadCsvUsers
Laravel will make a file under app/Http/Requests named UploadCsvUsers.php. Opening that file up, you will see:
<?php
namespace App\Http\Requests;
use Illuminate\Foundation\Http\FormRequest;
class UploadCsvUsers extends FormRequest
{
/**
* Determine if the user is authorized to make this request.
*
* @return bool
*/
public function authorize()
{
return false;
}
/**
* Get the validation rules that apply to the request.
*
* @return array
*/
public function rules()
{
return [
//
];
}
}
If you would like, read up more on these here. Change the "authorize" function to return true by default (if the function returns false it will stop the request and not run the rules). For now, just leave the rules empty and jump over to your routes/web.php file and add the form request to the route to see what happens. Your web.php file should now look similar to this:
use App\Http\Requests\UploadCsvUsers;
Route::post('/upload-new-users-csv', function (UploadCsvUsers $request) {
$uploadedFile = $request->usersCsvFile->openFile();
$returnArray = collect();
while (!$uploadedFile->eof()) {
$returnArray->push($uploadedFile->fgets());
}
return $returnArray;
});
Giving the code a run in phpunit, nothing's changed, you still fail. So, to avoid running off the rails here, you can check out all the available options for Laravel's validation here, but for now focus on the 'MIME Type By File Extension' rules. So, edit your Request and make it look like this:
public function rules()
{
return [
'usersCsvFile' => 'required|mimes:csv,txt'
];
}
Since you know the filename you want to use, use that as the key and add a couple rules here. The first one says the file is required for the upload, which is not being tested here but feel free to make a test of your own. Simply make a test that doesn't have the "usersCsvFile" in the upload parameters and you will see the request in action. The mimes rule here will accept a csv or text, since csv files are simply text.
Now if you run PHPUnit, you will get all green and you are officially done.
This is a pretty good test. Not only will this read CSVs like you want, but it also won't accept a file disguised as a CSV file.
Conclusion
When first uploading file tests and MIME type validation, it is easy to get thrown for a loop. This is especially true when one is used to using acceptance (browser) tests to do this, however the execution time of the test and effort needed to get those kind of tests built to do something that had very little (or nothing) to do with the actual user interface of the application was too high.
This is a way to do the testing faster and lighter. While this is by no means a fully explored example, it's a good place to start. Additionally, if you needed to do image uploads, with a little work, you could flip some things around and things should work the same.
About the Author
Terry Rowland is a Senior Web Application Developer at Enola Labs, an Austin, Texas-based web and mobile app development company.