My First Experience with GitHub Copilot X: A Programmer’s Perspective

Source

As a software engineer, I’m always on the lookout for tools and technologies that can help me work more efficiently. Recently, I had the opportunity to try out GitHub Copilot X, an AI-powered coding assistant that uses machine learning to suggest code snippets and autocomplete suggestions as you write code. I was excited to see what this new tool could do, and I was curious to see how it would integrate into my workflow. In this article, I’ll share my experience using GitHub Copilot X for the first time and offer my thoughts on whether or not it’s worth adding to your own toolkit.

I should note that for the purposes of this article, I decided to use only the features that are currently available under the public preview of GitHub Copilot X. Additionally, I used Visual Code as my coding environment and Go as my language of choice. Your experience with Copilot X may vary depending on your IDE and language, as some features may not be fully supported or integrated with certain IDEs. While there are other experimental or nightly build features that I could have tested, I wanted to provide a more accessible and realistic account of what it’s like to use this tool in its current state with a commonly used IDE. That said, I did explore the full range of features that are available within the public preview, and I was able to get a good sense of how they work and how they might be useful in real-world coding scenarios.

Integration into my IDE

Overall, integrating GitHub Copilot X into my IDE was a fairly straightforward process. Installing the extension was simple, but the authentication process was a bit annoying because Visual Code didn’t make it obvious where I had to authenticate with GitHub. Once I figured that out, however, the authentication process was relatively painless. I should note that you have to restart Visual Code before you can use the updated features, but this is a minor inconvenience at most. All in all, the overall process was not as difficult as I expected, and I was able to start using Copilot X in my coding workflow fairly quickly.

Writing New Code

The first and most obvious feature of GitHub Copilot X is its ability to assist with writing new, boilerplate code. In this section, we will explore how Copilot X can help you create new functions, classes, and other code structures from scratch. We will also examine how well Copilot X can adapt to your existing codebase and how much it can improve your overall coding workflow.

To test Copilot’s ability to generate new code that conforms to existing style, I decided to ask it to write a new function for a struct I use frequently in my coding work, called Snowplow, that I use to make queries against the Snowflake data warehouse. As a test, I decided to ask Copilot to write a new function that would retrieve financial assets associated with a date. I started by typing the function signature, including the receiver and name only: GetAssetsForDate. To my surprise, GitHub Copilot X immediately produced the following code:

func (client *Snowplow) GetAssetsForDate(ctx context.Context, provider gopb.Provider, date *time.Time) ([]*snowflake.Asset, error) {
 client.logger.Log("Getting assets data for %s provider...", provider)

 // First, create the query from our table name and select clause
 query := orm.NewQuery().Select(assetColumns...).From(client.fullName(Assets)).
  Where(orm.And, orm.Equals("provider_id", provider, false), orm.Equals("last_updated_utc", date, false))

 // Next, run the query and attempt to collect our results into a list of assets
 assets, err := orm.RunQuery[snowflake.Asset](ctx, query, client.db, client.logger)
 if err != nil {
  return nil, err
 }

 // Finally, return the assets we retrieved
 client.logger.Log("Retrieved %d assets for %s provider", len(assets), provider)
 return assets, nil
}

What really impressed me, however, was that Copilot was able to implement fully instrumented code similar to what I had already written in the same file. The comments associated with the code were also in language that I would have used.

Additionally, Copilot X utilized the same packages that I used to accomplish similar tasks elsewhere in that file. The code is syntactically correct and does what I would expect a function with that name to do. It takes in the proper data types and produces the proper output. After reviewing the code, I could find no obvious logical problems with it.

Moreover, Copilot was able to inject a type from the standard libary included with Go that I hadn’t imported explicitly, meaning that it has awareness outside of the specific file I’m working with. Overall, I was pleased with the quality of the code that GitHub Copilot X was able to generate based on my initial prompt.

To show just how close this suggested output is to what I normally write, I’ve included code from the same file as a reference:

func (client *Snowplow) GetAssetsForProvider(ctx context.Context, provider gopb.Provider, enabled bool) ([]*snowflake.Asset, error) {
 client.logger.Log("Getting assets data for %s provider...", provider)

 // First, create the query from our table name and select clause
 query := orm.NewQuery().Select(assetColumns...).From(client.fullName(Assets)).
  Where(orm.And, orm.Equals("provider_id", provider, false), orm.Equals("download_enabled", enabled, false))

 // Next, run the query and attempt to collect our results into a list of assets
 assets, err := orm.RunQuery[snowflake.Asset](ctx, query, client.db, client.logger)
 if err != nil {
  return nil, err
 }

 // Finally, return the assets we retrieved
 client.logger.Log("Retrieved %d assets for %s provider", len(assets), provider)
 return assets, nil
}

That being said, this output is not perfect. For example, the third parameter, date, is a pointer to a time.Time. This would be helpful if I wanted to make that parameter optional. However, the code produced by Copilot, given the ORM package I use, does not provide that capability.

Therefore, the user could potentially provide a nil argument to this function which would inevitably produce bad output. Another potential issue I noticed was that the function signature wasn’t commented. But, simply typing // on the line above the signature prompted Copilot to produce comments for the function, and those were of high quality as well.

Refactoring Existing Code

Refactoring existing code is an essential part of the software development process. Whether it’s improving performance, enhancing functionality, or simply making the code easier to understand, refactoring can have a significant impact on the quality and efficiency of your code. In this section, we will explore whether or not Copilot X can help you refactor existing code more quickly and accurately than your current process. We will also examine how well Copilot X can adapt to your existing coding style and how much it can improve your overall coding workflow.

To test Copilot X’s ability to refactor existing code, I experimented with several use cases, including transforming the data contained in an existing variable from one type to another, introducing non-breaking changes that require significant changes to existing code, and fixing a breaking change that resulted from a package update.

Transforming Data Types

In that same file, I have a list of column names that should be returned from asset-related queries:

var assetColumns = []string{"symbol", "provider_id", "asset_name", "asset_type", "market", "locale",
 "currency_symbol", "currency_name", "base_currency_symbol", "base_currency_name", "delisted_utc",
 "primary_exchange", "cik", "composite_figi", "share_class_figi", "last_updated_utc", "download_enabled"}

Now, suppose I wanted to convert this to a single, comma-delimited string. Can Copilot X do that? Yes, it can. Below this line, I began by typing var assetColums = “”. At this point, Copilot produced the following auto-completed output:

var assetColumns = "symbol, provider_id, asset_name, asset_type, market, locale, currency_symbol, currency_name, base_currency_symbol, base_currency_name, delisted_utc, primary_exchange, cik, composite_figi, share_class_figi, last_updated_utc, download_enabled"

Overall, Copilot was able to complete this task with minimal prompting, quickly generating a solution that was both syntactically correct and represented the same data. However, I did notice that in some cases where the length of the resulting line exceeded the norms exhibited elsewhere in the file, Copilot did not include newline characters.

While this may not be a significant issue for smaller code snippets, it could potentially lead to readability problems in larger projects, so it’s important to keep this in mind when using Copilot X for refactoring existing code.

Updating Existing Code

To test Copilot X’s ability to inject package updates, I attempted to replace raw string SQL queries with queries using an ORM package I imported.

First, I manually wrote an example query using the ORM package to give Copilot X an idea of what I was trying to do.

Next, I asked Copilot X to convert a raw string SQL query into ORM code. This is the code I started with.

querier := func(offset int, size int) *Query {
  return NewQuery(fmt.Sprintf("SELECT %s FROM %s WHERE provider_id = ? AND symbol = ? AND sip_timestamp >= ? "+
   "AND sip_timestamp < ? ORDER BY sip_timestamp LIMIT ? OFFSET ?", quoteColumns, client.fullName(Quotes)),
   provider, symbol, startTick, endTick, size, offset)
 }

After typing querier := func(o, Copilot X produced the following output with five tabs and enters.

querier := func(offset int, size int) *orm.Query {
  return orm.NewQuery().Select(quoteColumns...).From(client.fullName(Quotes)).
   Where(orm.And, orm.Equals("provider_id", provider), orm.Equals("symbol", symbol),
    orm.Equals("sip_timestamp", startTick), orm.Equals("sip_timestamp", endTick)).
   OrderBy("sip_timestamp").Limit(size).Offset(offset)
 }

While the output was partially correct, there were a few compiler and logical errors that needed to be fixed before the code would run. For reference, the correct code looked like this:

querier := func(offset int, size int) *orm.Query {
  return orm.NewQuery().Select(quoteColumns...).From(client.fullName(Quotes)).
   Where(orm.And, orm.Equals("provider_id", provider, false), orm.Equals("symbol", symbol, false),
    orm.GreaterThanOrEqualTo("sip_timestamp", startTick, false), orm.LessThan("sip_timestamp", endTick, false)).
   OrderBy("sip_timestamp").Limit(size, false).Offset(offset, false)
 }

After addressing these issues, I tried again with a different query, and this time Copilot X was able to generate the correct ORM code with no issues.

This experience showed me that Copilot X is not entirely aware of external packages or how they can be used but is capable of learning the syntax and usage quickly and even inferring the meaning of individual functions and variables.

In fact, later on, I attempted to convert a query that involved calling the Having function in the same ORM package, and Copilot was able to anticipate the existence of this function and what arguments it would take without me having to do anything.

Fixing Breaking Changes

As a developer, I’m curious to see if Copilot X is able to automatically fix compiler or logical errors introduced by breaking changes in a package update and if it can systematize the fix across multiple files. While Copilot is a powerful tool for generating new code, it’s unclear if these capabilities extend to fixing breaking changes in packages.

In this section, I’ll test this hypothesis by intentionally introducing breaking changes in a package and seeing if Copilot is able to identify and suggest fixes for the affected code.

This will provide valuable insights into the tool’s capabilities and limitations, and may help other developers make informed decisions about how to integrate it into their own workflows.

To test this, I added an argument to an interface and a single function and then imported the updated package into my main module. When attempting to provide a constant string for the missing argument, Copilot was unable to anticipate the value of the string or provide any suggestions after two successive tries across different files.

Although the sample size for this operation is small, it is similar to the learning rate exhibited by Copilot in other instances. This suggests to me that either Copilot is incapable of anticipating small changes of this nature without more examples or else the test itself might have been too limited for Copilot to show its true potential. That being said, rather than suggesting options that were obviously wrong, Copilot refrained from suggesting anything at all. Although I would find this lack of suggestion annoying from something like Intellisense, I respect Copilot’s ability to avoid providing garbage output when it lacks confidence in its suggestions. I will have to continue testing in this particular area to identify exactly where it fails and where it provides useful suggestions.

Generating Tests

In this section, I will explore Copilot’s ability to generate test code. Specifically, I want to determine if it can generate full test suites, infer how tests should be structured based on testing libraries or frameworks, and generate valid, realistic test data.

Additionally, I want to see if Copilot can find corner cases or common failure modes based on the type of test and determine the expected responses or values of generated data. By examining these aspects of Copilot’s functionality, I hope to gain a better understanding of how it can assist with software testing and where its limitations may lie.

To that end, I asked Copilot X to attempt to generate unit tests for the function we had it generate earlier: GetAssetsForDate. In order to avoid code reuse, and to make testing easier, our tests rely on the gomega and ginkgo packages for assertions and test structure and we use go-sqlmock for testing SQL code, specifically. Moreover, the suite these tests will be added to contains two types: tests for paged query functions and tests for single query functions.

As this function contains a singleton SQL query, the test should have a structure similar to the former rather than the latter. Additionally, the tests involve several different conditions including bad data conditions and SQL error conditions. Finally, when errors do occur, we rely on an internal package to format the error and surface additional information such as the environment, package, class (if available), function, inner error and message. Will Copilot be able to generate test code that conforms to all of these requirements?

To generate the test code, I first wrote a comment describing the desired behavior. Copilot then began generating the test code, line by line. Initially, it attempted to generate a test for a paged query function, likely because I had been working on that type of test before writing this post. However, it was still able to generate much of the boilerplate code that I would have otherwise copy-pasted, and it created test data that was consistent with what I use in other tests within the same file. While it did generate some different test data initially, it may have learned that I tend to reuse the same data across multiple tests for efficiency.

However, Copilot was unable to generate the exact SQL query that the function would have sent to the server, but the differences were easy to fix. When generating the function call, Copilot tried to provide a string argument for the time.Time instead of creating a call to time.Parse or time.Date. Additionally, it required assistance in creating the appropriate error verification code, but it was able to learn my expectations after a few iterations. It even generated the correct Go type errors without any input from me, which was helpful. After making some adjustments, I ended up with the following code:

 // Tests the conditions determining which errors are returned from the GetAssetsForDate function when it fails
 DescribeTable("GetAssetsForDate - Failures",
  func(queryFails bool, rowErr bool, scanFails bool, verifier func(*utils.GError)) {

   // First, setup the mock to return the rows
   svc, mock := createServiceMock("TEST_DATABASE", "TEST_SCHEMA")

   // Next, create the rows that will be returned by the query
   rows := mock.NewRows([]string{"symbol", "provider_id", "asset_name", "asset_type", "market",
    "locale", "currency_symbol", "currency_name", "base_currency_symbol", "base_currency_name",
    "delisted_utc", "primary_exchange", "cik", "composite_figi", "share_class_figi",
    "last_updated_utc", "download_enabled"}).
    AddRow("AAPL", 1, "Apple Inc.", 1, 4, 0, "USD", "", "", "", "", "XNAS",
     "0000320193", "BBG000B9XRY4", "BBG001S5N8V8", "2022-06-20T00:00:00Z", false).
    AddRow("AAP", 1, "ADVANCE AUTO PARTS INC", 1, 4, 0, "USD", "", "", "", "", "XNYS",
     "0001158449", "BBG000F7RCJ1", "BBG001SD2SB2", "2022-06-20T00:00:00Z", false).
    AddRow("AAUKF", 1, "ANGLO AMER PLC ORD.", 2, 5, 0, "USD", "", "", "", "", "",
     "", "", "", "2022-03-08T07:44:15.171Z", false).
    AddRow("BA", 1, "Boeing Company", 1, 4, 0, "USD", "", "", "", "", "XNYS",
     "0000012927", "BBG000BCSST7", "BBG001S5P0V3", "2022-06-20T00:00:00Z", false).
    AddRow("BABAF", 1, "ALIBABA GROUP HOLDING LTD", 2, 5, 0, "USD", "", "", "", "", "",
     "", "", "", "2021-02-11T06:00:50.792Z", false).
    AddRow("BAC", 1, "Bank of America Corporation", 0, 0, 0, "USD", "", "", "", "", "XNYS",
     "0000070858", "BBG000BCTLF6", "BBG001S5P0Y0", "2022-06-20T00:00:00Z", false)
   if rowErr {
    rows.RowError(0, fmt.Errorf("row error"))
   } else if scanFails {
    rows.AddRow("BAC", "derp", "Bank of America Corporation", 0, 0, 0, "USD", "", "", "", "",
     "XNYS", "0000070858", "BBG000BCTLF6", "BBG001S5P0Y0", "2022-06-20T00:00:00Z", true)
   }

   // Setup the mock so we can verify the SQL query when it is submitted to the server
   queryStmt := mock.ExpectQuery(regexp.QuoteMeta("SELECT symbol, provider_id, asset_name, "+
    "asset_type, market, locale, currency_symbol, currency_name, base_currency_symbol, "+
    "base_currency_name, delisted_utc, primary_exchange, cik, composite_figi, share_class_figi, "+
    "last_updated_utc, download_enabled FROM TEST_SCHEMA.assets WHERE provider_id = ? AND "+
    "last_updated_utc = ?")).WithArgs(gopb.Provider_Polygon, time.Date(2021, time.January, 1, 0, 0, 0, 0, time.UTC))
   if queryFails {
    queryStmt.WillReturnError(fmt.Errorf("query error"))
   } else {
    queryStmt.WillReturnRows(rows)
   }

   // Finally, call the function and verify the error
   assets, err := svc.GetAssetsForDate(context.Background(), gopb.Provider_Polygon,
    time.Date(2021, time.January, 1, 0, 0, 0, 0, time.UTC))

   // Verify the error and the returned assets
   verifier(err.(*utils.GError))
   Expect(assets).Should(BeEmpty())
   Expect(mock.ExpectationsWereMet()).ShouldNot(HaveOccurred())
  },
  Entry("Query fails", true, false, false,
   testutils.ErrorVerifier("test", "snowplow", "/xefino/quantum-core/pkg/snowplow/assets.go",
    "Snowplow", "GetAssetsForDate", 86, testutils.InnerErrorVerifier("Failed to query data from "+
     "the TEST_DATABASE.TEST_SCHEMA.assets table"), "query error", "[test] snowplow.Snowplow.GetAssetsForDate "+
     " (/xefino/quantum-core/pkg/snowplow/assets.go 86): Failed to query data from the "+
     "TEST_DATABASE.TEST_SCHEMA.assets table: query error")),
  Entry("rows.Err fails", false, true, false,
   testutils.ErrorVerifier("test", "snowplow", "/xefino/quantum-core/pkg/snowplow/assets.go",
    "Snowplow", "GetAssetsForDate", 86, testutils.InnerErrorVerifier("Failed to query data from "+
     "the TEST_DATABASE.TEST_SCHEMA.assets table"), "row error", "[test] snowplow.Snowplow.GetAssetsForDate "+
     " (/xefino/quantum-core/pkg/snowplow/assets.go 86): Failed to query data from the "+
     "TEST_DATABASE.TEST_SCHEMA.assets table: row error")),
  Entry("rows.Scan fails", false, false, true,
   testutils.ErrorVerifier("test", "snowplow", "/xefino/quantum-core/pkg/snowplow/assets.go",
    "Snowplow", "GetAssetsForDate", 86, testutils.InnerErrorVerifier("Failed to query data from "+
     "the TEST_DATABASE.TEST_SCHEMA.assets table, error: sql: Scan error on column index 1, name "+
     "\"provider_id\": converting driver.Value type string (\"derp\") to a int64: invalid syntax"),
    "[test] snowplow.Snowplow.GetAssetsForDate (/xefino/quantum-core/pkg/snowplow/assets.go 86): "+
     "Failed to query data from the TEST_DATABASE.TEST_SCHEMA.assets table, error: sql: Scan error "+
     "on column index 1, name \"provider_id\": converting driver.Value type string (\"derp\") to a "+
     "int64: invalid syntax")))

While Copilot was able to generate some of the boilerplate for a test function, it struggled to infer the test structure based on the testing libraries and frameworks in use. Additionally, it had difficulty generating realistic test data and finding corner cases or common failure modes.

However, it was able to learn from previous iterations and generate valid Go type errors without input. Overall, while Copilot shows promise in generating test code, it is not yet a comprehensive solution for generating tests. Developers should still be prepared to manually review and modify the generated code to ensure comprehensive test coverage.

Conclusion

Copilot is a powerful tool that can enhance the coding workflow of experienced developers by generating new code or by copying patterns from existing code. It is especially helpful when there is a consistent pattern for existing code in the same file. However, it requires some examples before it can generate anything that should integrate with anything else. Copilot generates syntactically correct code about 95% of the time, but it still has trouble referencing types across different files or in different packages.

The API is still pretty slow (~500ms per line), and it’s not particularly good at generating tests that would pass, although this may be useful from a TDD perspective. Despite its limitations, Copilot is a useful tool for experienced developers who want to be more efficient at work. However, junior developers and beginners in computer science should not rely on this tool as it lacks the ability to instill best practices specific to the language it’s running on. Overall, I’d say that Copilot’s billing as an AI pair programmer is about half-correct: it can generate code while you review, but it is incapable of fulfilling the reviewer role.

If you’re a senior developer looking to optimize your workflow, I highly recommend giving Copilot a try. It can help you generate new code and patterns quickly and efficiently. However, keep in mind that it’s not a replacement for good coding practices, and it’s not suitable for beginners or junior developers. As AI and machine learning continue to advance, we’re sure to see even more exciting developments in this field. So, stay curious, keep learning, and let’s see what the future holds for AI and programming!