Building an Online Retail Store – Update 2

I’ve been looking at what extensions I’ll need for my online store. A good SEO module is unavoidable, as I need SEO-friendly urls for product links and easy access to meta data, as well as the ability to generate it. I’ve found just the extension I need that does all this, so I’ll be purchasing it soon. The second extension I’ll be getting is for improved search functionality. Auto-complete suggestions, much like google has when typing in a search, is pretty useful for product searching, and this is what the second extension will do. After that, any further extensions will be optional, but I feel these two are absolutely essential for any modern online store.

I’ve reinstalled the store on a new domain, so I’ll just use the current one for database testing/breaking. I’m making regular backups, as although I haven’t broken anything (yet), anything I do break may be irreversible. I’ll be using my test store for product importing until I’ve perfected my import system.

I’ve added Paypal as a payment option to the store, and I’m considering whether to also use Paypal as a Credit Card Processor, or to install a separate payment system to handle Credit Cards, such as SagePay. The latter would incur a small monthly cost, whereas Paypal charge you fees, so the sensible option would be to use Paypal to begin with to keep overheads to a minimum, and then consider switching to a system that charges monthly when the costs of doing so are less than the fees Paypal charge for Credit Card Processing.

I’ve enabled CloudFlare on my online store, which means part of the website will be stored “in the cloud”. Basically what this means is that CloudFlare stores a cache of the pages of the store, and shares some of the bandwidth load. This, theoretically, results in better load times and it should definitely result in reduced bandwidth. My only reservation with it so far is that if I make any changes to the site, in order to see those changes, I have to manually submit the page url to CloudFlare so it flushes its cache of the page.

You essentially have two copies of the website. Your copy, and the copy on CloudFlare’s servers. It’s not a good idea to have it enabled when you’re making a lot of changes to your website. You can force CloudFlare to re-cache your entire website, but this is slow, and will eat up your bandwidth. For now, I’m going to disable it, and re-enable it before the website launches. Overall I think the benefits are worth its use, especially when my web hosting provider includes CloudFlare with my hosting package at no additional cost, so I may as well take advantage of it, if only to reduce my monthly bandwidth usage.

I’ve also set up a completely new store theme. It’s clean, and very easy to navigate. I’ll spend some time tweaking the layout after I’ve imported all the products, which is what I’m currently working on.

OpenCart, unfortunately, doesn’t natively provide an extension for the bulk importing of products, and the free extension I tried seems to be broken in version of OpenCart. This leaves me with two options; buy an extension, and hope it’s flexible enough to map all the product specifications and images, or write my own script in PHP to import the products in bulk. I’m of the opinion that the latter would be the best approach, as although it may initially take longer to have a workable solution, it will allow a much greater deal of flexibility in future, should I need to add more products. In the long run, that is what will be most beneficial.

I’m currently working on obtaining all the product images for the products I’ll be listing in my online store. I’ll be selling PC components, fans, computer cables, CPU coolers, mice, keyboards, and products of that nature. Before I can work on the main product importer script, I have to obtain high quality images for all the products, which I’ve written a separate script for. The process of obtaining the images will be in two stages;

1) Run the script to find and download the product images, putting these links in the product feed file for use later in the script for importing products into the website
2) Downloading all these images in bulk from those links

I’ve already written code to accomplish step 2 as well.

As you can see, writing scripts on the fly to accomplish tasks that would otherwise take considerably longer or be prohibitively expensive on a small budget is extremely advantageous. Of course, not every problem is best solved by writing a script, so one must always consider the time, the cost and reward of doing so.

Building an Online Retail Store – Update 1

I decided to avoid the Softaculous Script Installer that comes with my cPanel, and downloaded and installed OpenCart manually from their website instead.

It was a relatively painless experience. First I created a sub-domain for my store, then I installed the files to the FTP location associated with the new sub-domain, created a new database for the store, followed the installation instructions, and that was pretty much it.

Naturally, at the time of making this post, this is just a fresh OpenCart installation. I’ll probably experiment with changing the theme and installing extra extensions first as I familiarise myself with OpenCart. I’ll likely move this over to a new domain once it’s in any kind of fit and usable state, but for now, I have a usable website to write scripts for.

I’ve shortlisted a few domain names, as I’ve already settled on a product range should I ever decide to launch the store. There’s a lot of work that needs to be done before even considering that, though. If I ever decide to launch it at all, that is.

For the moment, I’m keeping the store in maintenance mode. Wouldn’t want anyone to think this is an actual live store. The site right now looks identical to this demo from OpenCart’s website. I’ll bring the site back up once I figure out how to disable registration.

Next I’ll create some dummy orders to see how they’re stored in the database orders table, and see how much work will be involved in importing and creating several hundred products from a spreadsheet.

Building an Online Retail Store – Development Log

These logs will mainly outline my progress as well as the steps required to manage multiple online retail stores via a single database, using a custom-designed series of systems to automate a lot of tasks that would be otherwise take up too much manual labour.

These include;
– Automating order imports
– Automating order tracking updates
– Automating stock levels
– Automating pricing

To handle this, a set of systems need to be put in place that manage the stock levels, prices and orders across multiple channels, and store them in one place. One solution would be to rely on systems already developed such as ChannelGrabber or ChannelAdvisor, however there are disadvantages to this, in case of problems with those systems, and being limited to the tools those systems provide. Instead, having a set of systems in-between a system such as ChannelGrabber and OpenCart to manage stock levels will allow a greater level of control, while maintaining the use of the functionality of both, and avoiding creating a single dependency on one system.

The store will use a central database to manage stock levels and pricing across multiple store channels, and act as a central storage for pricing, stock, orders and customer data. So, the overall aim will be a centralised management system that can manage an online retail store, and additional shop channels such as Ebay, Amazon, Rakuten, and be expanded upon in future. A lot of the code work has already been dome, but as these systems were built previously for an X-Cart website, the database script will need to be modified to conform to the database structure used by OpenCart. I’ve also also written an order importer that can import orders from ChannelGrabber into a database, but again, this was designed for X-Cart, so this will also need to conform to OpenCart’s database structure.

However, before all of these things can be looked at, I must first install a test version of OpenCart so I can take a look at the database stucture. The first script I may need to write would be one which will turn a .csv product feed into products to populate the store with.

An FTP wrapper class for file handling in PHP

Some time ago, I had to develop a system for handling a large amount of CSV files that were automatically being uploaded to an FTP location. As it was an automated system, I needed a class flexible enough to handle a lot of the common ftp functions I would be making use of. It also contains a couple of additional functions that were more specific to my needs. The latest_file method will take the latest file from the current directory based on the date the file was last modified. This was pretty essential to my needs at the time as the directory was being filled with new CSV files on a regular basis and were irregularly named, so I had to ensure I was only taking the latest file.

The class is also able to make several connection attempts to an FTP, which you can modify using the methods set_attempts and set_delay, which sets the delay between attempts in seconds.

Here’s a basic usage example;

Here is the entire class;

Filter bad input characters using whitelists in PHP

When working with input data using PHP and storing it using MySQL, it can be a bit of a nightmare to filter out bad characters. Any time I’ve had to work with a large dataset full of customer data or product data, it was inevitable that there would be invalid characters, either because it wasn’t supported by the charset of the database, or because whatever system created the original datafile file didn’t support the characters, leading to garbage text. When dealing with customer data it was important not to lose any information during the importing process. This usually involved a conversion table of one kind or another, converting the invalid characters to friendly equivalents so they could be viewed correctly. Especially important, for example, when dealing with customer addresses. Even if it was possible to store the original characters, many third-party shipping services didn’t support them anyway.

So for customer data this was fine, but where this manual approach wasn’t very practical was when creating a large retail product database, which then also had to be uploaded to a third-party platform to list the products in other e-commerce stores. The product data was obtained from a variety of difference sources, which further increased the likelihood of compatibility issues. Time and time again, there would be an unusual character somewhere in the data that would break the entire product description when it was uploaded to the third-party platform. I traced the problem down to the third-party’s MySQL query, which wasn’t robust enough to handle unsupported characters. It broke the query right at that point the invalid character was found, and the rest of the data wasn’t uploaded. Because this data also included a html template for the product description, it led to some very broken product listings. It wasn’t possible to fix this problem at the source, so I had to look at ways of sanitising my data.

Given the amount of products that were being uploaded, it simply wasn’t practical to manually correct each one, especially when there was no real way of knowing which product listings were correct and which were broken due to invalid characters, and in almost every case it was just one stray invalid character that didn’t have any impact on the product description. The conversion table option was also out, as this would have taken just as long. Given there were time constraints, this gave me two options; a blacklist of invalid characters, or a whitelist of supported characters. A blacklist seemed just as much work, as it would still require finding all of the invalid characters, and also dealing with the issue of finding them correctly in PHP, which is rarely straight forward when dealing with different charsets. This can involve needing to find the hex code of a lot of invalid characters in order to replace them. This left me with the option of using a whitelist; a list of characters I knew would not break the third-party’s MySQL query, and would result in almost negligible data loss in the product listings.

This is what I came up with;

This function takes in a whitelist string and an input string. Anything not in the whitelist is removed, and then the string is returned by the function with only the characters from the whitelist remaining. For ease of use with my script, I stored my whitelist string in an external .txt file so it could be edited when needed. This was the whitelist I used;

    ! “#$%&'()*+,-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

Which was sufficient in removing the stray unsupported characters from the imported product listings without having any major impact on the quality of the data. While not a perfect solution, it is a good compromise, and additional characters can be added or removed from the whitelist to refine it over time.

Obtain the first or last word inside a string in C++

If you do a lot of work with strings in C++, one thing you’ll probably find yourself doing is writing utility functions. The String class provides many useful built-in methods, but when it comes to searching, replacing and text manipulation, you may find that you’ll need to create your own functions to carry out these tasks, sometimes using and combining the built-in class methods to carry out more complex tasks within your projects.

The two following functions can be used to obtain the first and last words contained within a string. An empty string is returned if the function fails. The functions make use of the built in String methods find_first_of and find_last_of;

Having a decent set of your own utility functions can save a lot of time, and will make your code a lot more manageable. A good strategy for code re-usability is to break up a bigger task into smaller functions that you can see yourself re-using in the future, saving yourself both time and making your code easier to understand.

This is akin to your toolbox, and if you find yourself needing the same tool over and over again, then that’s when it’s a good idea to write your own utility function or use someone else’s. There’s always that temptation to write everything for yourself, but one of the key aspects of code re-usability is to not re-invent the wheel.

Writing your own utility functions does have some benefits, however. Firstly, you’ll understand the code very well and learn new techniques, and secondly, it can improve your overall skills as a programmer, especially in problem solving. So yes, there are times when it is better to write your own, and other times when using someone else’s solution is the best approach. You may even decide to compromise by checking other people’s solutions to help you write you own. There’s no right or wrong way to approach it. Take the approach that suits you or the situation best.

Digital Cityscape – Fractal Art

Digital Cityscape - Fractal art

Digital Cityscape – Fractal art

Created using Ultra Fractal 5.04. You can see a higher quality render of the artwork here or by clicking the image above.

Contains 6 individual layers in total, utilising one fractal formula and three different colouring algorithms. Might experiment with this particular fractal formula again in future, as it has some interesting properties for geometric fractals.

Prime numbers, a simple function in C++

A prime number is simply a number that is only divisible by one and itself. One of the most basic ways to find out if a number is prime, then, is to loop through every number (except for 1, as all positive integers are divisible by 1), and return true if the number is equally divisible by any other number except for itself.

This is a good example use of the modulus operator. The modulus operation returns the remainder of a division of two numbers. So, for example, 15 % 5 would equal 0, as 5 goes into 15 exactly 3 times with no remainder. 17 % 5 on the other hand would equal 2. As 5 goes into 17 3 times, with 2 being left over.

How it works:

Knowing this, and knowing what makes a prime number, we can check if a number is prime by checking every number between two and itself minus one, and if any of these numbers give us a modulus result of 0, then it is exactly divisible by another number and therefore not prime. If no modulus results of 0 are found by the time i is equal to x-1, then we know x to be prime. As each integer would always be divisible by itself, we have no need to check whether this is true or not. If we did want to check this, then our loop conditional statement would be i <= x;

This type of function is known as a predicate. A predicate is simply a function that returns true or false. In this instance, given an input of x, it will return true if the number is prime, and false if it is not.

If you wanted to implement this function and check every number beyond 1 to see if it was prime, you could do it like so;

This will iterate indefinitely until the maximum value that is able to be held in unsigned long int on your machine is reached. At that point, it will wrap around back to 0. You can use a larger data type, such as long long int, if your compiler supports it. But as you'll see, the main drawback of this function is that the larger the number gets, the more numbers it needs to check itself against to determine whether or not it is prime, so you may never reach that limit as it will take progressively longer each time to check the next number in the infinite while loop. The speed at which the program runs will depend on the processing power of the machine you are running it on.

If you needed to quickly check if a number is prime without needing to calculate it, then you could store the results in a text file, and check to see if that number exists in the file. If you store the results numerically, then you can stop searching once you reach the next prime number in the file beyond the one you're searching for. This is a very simple example of a lookup table, which saves computation time by storing the results of an operation rather than having to re-calculate them each time. This makes sense when you're performing a large number of checks of a particular calculation where the result is always going to be the same, and especially if you're going to be performing the same calculations many thousands of times over, and performance is a factor.

There are much more efficient techniques out there for finding prime numbers too, known as prime number sieves. One such algorithm is known as the Sieve of Eratosthenes, and I may follow up this post in future with a discussion on that.

Permute all possible string combinations in C++

This function takes in a string, and will permute every possible combination of arrangements of the characters in the string. Because the function simply iterates through every character, it can output duplicate results. In the case of the word “sells”, for example, the first and last s will be swapped, resulting in the same word being output. One lazy way around this would be to store each combination as a key -> value pair using the STL container class map, which would discard any duplicate keys (although it may be more accurate to say that it would replace the value stored at that key, which would be the duplicate word). Another way would be to use the STL container class vector with std::find, however with larger words this may be a less performance-friendly approach than using a map.

I wrote this function in response to this programming challenge. Using the given input of “cat”, this function actually outputs:


So, it doesn’t replicate the example output given by the challenge exactly, (same results, different order) however, the requirements were to find all permutations, and repeated outputs in the case of duplicate characters are allowed too, so the function satisfies the challenge requirements.

How it works:

The function contains two loops, with one nested within the other. The outer loop iterates through a single character at a time, executes the inner loop, and then swaps each character with the next character at index i+1. The inner loop takes each character from the back of the string, moves it to the front, and then prints the string to the console. Combining these two processes allows us to permute every possible combination of characters, and the function will work on any size string.

You’ll notice that the outer loop iterates while i is less than str.length()-1, rather than str.length(). Why? Because we’re swapping the character at position i of the string with the character at position i+1 at the bottom of the outer loop, we have to ensure we do not go out of bounds of the string. If we used str.length() instead, then on the last iteration of the outer loop when we assign str[i] = str[i+1];, we’d be attempting to access an index just beyond the length of the string, which would give us undefined behaviour. It could crash the program, access some other data stored in memory, or do nothing at all. In all likelihood it would eventually crash the program.

The outer loop will run while i is less than str.length()-1, so it will stop executing once i is equal to the last index location of the string, preventing us from attempting to access an element beyond the bounds of the string.

We also use str.length()-1 to ensure we assign the last character in the string to buf within the inner loop. The length method of the string class gives the total number of characters in the string, but as the index of a string starts at 0, the last index in the string is equal to str.length()-1

If you’re still not entirely sure how the function works, then try it for yourself. Try commenting out the inner loop entirely and running the code, or see what happens when you don’t swap the character at position i in the string with i+1 in the outer loop.