Tips for Dealing With External Services
... we all don't like them but have to deal with them π
I don't like external services, as most developers do, but businesses like them. For good reasons (on both sides).
The inherent issue is that they are out of our control.
Here are some of the common issues with external services
Lack of documentation - every system is different
Dealing with API keys
Unexpected errors
Performance issues
API throttling - too many calls and they kill requests
API pricing - usually you pay for the service, but also recently there is a new trend of paying for API access
How to integrate with the application
Lack of safe sandbox playground - often, you have to call the life system to test while developing.
Unexpected changes of those APIs
In my career, I have integrated with many external services. Here are some of my tips how to deal with them.
Let's break these down.
1/ Keep external services code in a single place π
AllΒ external services live in a single place.
If I use Ruby on Rails, I have an `external
` module, living in `app/lib/external/[service].rb
`.
If I'm using React, they all live in `utils/external/[sercice].ts
`.
This even includes code for microservices I control.
By adopting this practice, I can efficiently monitor most of the outgoing network calls from my application by simply looking at this folder.
2/ Wrap external services code in a facade π
I ALWAYS wrap all external services with a facade.
Even when they have their libraries.
In this way
I know exactly what is being used from the service
Updating service code or even replacing service is easier
It is easier to test our code because I can stub the service facade.
There is a single place, to put documentation
Makes upgrading the service API very easy since you know where its impact points are
Ruby example
I use `extend self` facade module in Ruby, named `External::[Service]Api
`. Usually, those are just bags of methods.
Here is a template for external services:
# frozen_string_literal: true
# Documentation
#
# - service: [url to service]
# - manage: [url to manage tokens]
# - api: [url to api documentation]
# - gem: [gem we are using (optional)]
# [- ... other links]
module External::[Service]Api
extend self
# documentation: [documentation]
def perform_action(args)
# use HTTParty or gem for this external service
end
end
Here is an example for using Unsplash api:
# frozen_string_literal: true
# Documentation
#
# - service: https://unsplash.com/
# - api: https://unsplash.com/documentation
# - gem: https://github.com/unsplash/unsplash_rb
# - portal: https://unsplash.com/oauth/applications
module External::UnsplashApi
extend self
# documentation: https://unsplash.com/documentation#search-photos
def search(query)
Unsplash::Photo.search(query, 1, 12, 'landscape')
end
# documentation: https://unsplash.com/documentation#get-a-photo
# documentation: https://unsplash.com/documentation#track-a-photo-download
def track_download(id)
photo = Unsplash::Photo.find(id)
photo.track_download
rescue Unsplash::NotFoundError
nil
end
end
Notice all the links to the documentation here.
Even when the external service has a gem to access it, the gem is never used outside this module.
If the gem returns an instance of a response object, I try to wrap it with a class I own. I expose only what I use from the services.
JavaScript example
I follow the same rules in rules in JavaScript. There, instead of a module, I export functions. I still have links to the documentation in the comments.
Here is an example of an Intercom facade:
let identified = false;
export function identifyUser(userInfo) {
if (identified) {
return;
}
if (userInfo.userId) {
runIntercom('onShow', () => {
window.Intercom('update', userInfo);
identified = true;
});
}
}
export function openMessage(message) {
runIntercom('showNewMessage', message);
}
export showProductTour(tourId) {
runIntercom('startTour', tourId);
}
function runIntercom(cmd: string, args: any) {
if (typeof window === 'undefined' || !window.Intercom) {
return;
}
return window.Intercom(cmd, args);
}
External JavaScript libraries often depend on injecting scripts. I try to include those in `external
` as well.
For example, exporting the Intercom script tag lives as `utils/external/intercom/script
`.
// Documentation: https://app.intercom.com/a/apps/[code]/settings/web
export default `
(function() {
window.intercomSettings = {
app_id: "[code]"
};
(function(){var w=window;var ic=w.Intercom;if(typeof ic==="function"){ic('reattach_activator');ic('update',w.intercomSettings);}else{var d=document;var i=function(){i.c(arguments);};i.q=[];i.c=function(args){i.q.push(args);};w.Intercom=i;var l=function(){setTimeout(function(){var s=d.createElement('script');s.type='text/javascript';s.defer=true;s.src='https://widget.intercom.io/widget/[code]';var x=d.getElementsByTagName('script')[0];x.parentNode.insertBefore(s,x);}, 4000);};if(w.attachEvent){w.attachEvent('onload',l);}else{w.addEventListener('load',l,false);}}})();
})()
`;
3/ Store credentials to external services with linksπ
The key concepts here are
Don't put credentials in plain text.
Except when you expose API keys in frontend, those will be available in frontend anyway
Have credentials in a centralized placed
Have links where those credentials come from
Have comments who have added those credentials
In Ruby on Rails, I use encrypted credentials.
Credentials are stored in YAML. I always include a link to where a certain credential was taken because it is very irritating to search where a token and secret were taken, especially when dealing with Google APIs.
# Taken from: [url to where those credentials are taken]
[sevice_name]_token: [...]
[sevice_name]_secret: [...]
I like to group all those credentials in an `extend self module
` named `Config
`, along with other app configurations.
Other technologies use primary ENV (Environment) variables for credentials. In this case. What I do is
Access ENV variables only from a single config file
...or two if I have server and client-specific ENV variables
Have links to where credentials were taken in this config file
I can't stress enough how important it is to have a link and explanation of where a certain API key comes from.
It is very frustrating to wonder if this is the correct key. Has it expired? Which app uses it?
4/ Keep a log of calls to external services in a database π
This one depends a lot on the service call. If I make external calls to something like Stripe, I only keep the `_external_id
` returned by Stripe.
In Angry Building, I leverage Twilio to send SMS to users. I maintain a database table named `external_twillio_messages
` to keep track of every message and its deliverability. This helps me monitor the effectiveness of my SMS campaigns and aids in cost management, as Twilio charges are based on the number of SMS messages sent.
I called Twitter API to get tweets in the past, and I only needed immutable data from them. Twitter had hard limits in terms of API rating. Having a database table `external_twitter_tweets` to store tweets helps me only to fetch data once.
5/ Perform calls to external services in background jobs βοΈ
Instead of making service calls directly in your application during web requests, you move to a background job. Those jobs should be idempotent.
When you call external services, you're making network calls. By moving these calls out of your application's main execution path and into background jobs, you are optimizing your application's performance.
In this way, if those services are down or there are network issues, your background job system will handle them.
You can combine with my previous advice and have a database table for external calls and have a flow like the following:
Create a record in the database with the pending status
Trigger background job
When the background job is done, mark the record as complete
Notify the user when completed
Refresh UI
Send notification
Conclusion
When you are dealing with external services, you should be very careful and isolate them from the rest of your system.
In this way, maintaining those external services is a lot easier.
Apart from that, I suggest leaving many comments with links to documentation about this external API.
If you have any questions or comments, you can ping me onΒ Threads,Β LinkedIn,Β Mastodon, Twitter or just leave a comment below π