Critical Challenges

Scholarly, research and creative activities

In the absence of adequate guidance, training, and resources, AI technologies can present significant challenges to research integrity. There are several elements of the research enterprise where AI tools may undermine research integrity. These include the development and assessment of research grant applications, peer review, communication of research findings, data validation, research ethics, intellectual property protections, and the creation of creative works.

While AI tools have potential to aid researchers in the development of grant proposals, reviewing drafts and providing feedback, they may also undermine that process if researchers offload critical cognitive and analytical work to machine assistance. There is significant risk of the inadvertent inclusion of factual errors in AI-generated grant proposals as it is well established that LLM-based AI systems that fabricate plausible text can generate inaccurate information. External funding agencies have already established AI guidelines that TMU researchers should consult when preparing grant applications. For example, the Canadian research funding agencies have published specific guidance concerning the use of AI in grant proposals. While it recognizes that “generative AI may be a valuable tool to applicants in the preparation of grant applications, including the potential to improve efficiency, assist non-native English and French speakers, and streamline the proposal writing process,” it also cautions that “applicants are responsible for ensuring that information included in their grant applications is true, accurate and complete and that all sources are appropriately acknowledged and referenced.”⁹ Other funding agencies, such as the National Institutes of Health, prohibit substantial use of AI and “will not consider applications that are either substantially developed by AI, or contain sections substantially developed by AI, to be original ideas of applicants.”¹⁰ Furthermore, most guidance prohibits the use of publicly available online AI tools for the review of grant applications as they may breach conflict of interest and confidentiality agreements.  

There are similar challenges in the communication of research findings through scholarly publication and peer review. Academic journals have been struggling to confront the complicated challenges associated with both AI-generated manuscripts as well as the use of AI in peer review. To some extent, both applications of AI may undermine the academic enterprise. Most journals have established their own guidance concerning the use of AI in publications and many prohibit or severely limit the use of AI in peer review. For example, Journal of the American Medical Association precludes the inclusion of nonhuman machine authors and requires transparent disclosure of the use of AI tools in the preparation of manuscripts for publication. It also cautions authors that they are responsible for the integrity of all content submitted for publication. Its editors further limit the use of AI in peer review, noting the high risk for inaccuracies and confidentiality breaches.¹¹

Peer review is an especially sensitive domain for AI application. In a study of the implications of the use of LLM technologies in peer review, Hosseni and Horbach found,

LLMs have the potential to substantially alter the role of both peer reviewers and  editors. Through supporting both actors in efficiently writing constructive reports or decision letters, LLMs can facilitate higher quality review and address issues of review shortage. However, the fundamental opacity of LLMs' training data, inner workings, data handling, and development processes raise concerns about potential biases, confidentiality and the reproducibility of review reports. Additionally, as editorial work has a prominent function in defining and shaping epistemic communities, as well as  negotiating normative frameworks within such communities, partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia.¹²

Given the significance of these challenges and limitations to the use of AI tools in the preparation of scholarly manuscripts and peer review, it will be important for the TMU research community to remain engaged in these issues both within the university and across specific disciplinary communities and contexts.

Just as AI tools used in peer review raise concerns about confidentiality and privacy, so too do the use of these tools in research methods. AI transcription tools, for instance, have the potential to improve and expand the capacity to process audio recordings from interview subjects, but the use of such tools must be subject to rigorous research ethics review. Most publicly available consumer AI chatbots collect user input to further train LLMs and thus present significant data privacy risks. Use of locally hosted and open source models may offer more ethical and responsible applications of AI in research methods.

The legality of using copyrighted material to train or prompt LLM-based AI tools is a rapidly evolving area of law currently being tested in the courts. At the heart of the debate is whether using copyrighted material as AI input constitutes fair dealing. This remains an area of uncertainty and risk assessment in Canada. Courts in the U.S. have held that training of LLMs is a highly transformative use of existing content.¹³ Accordingly, the Library Copyright Alliance in the US has stated its position that the training of an LLM using existing works could be considered fair use.¹⁴ With fewer court precedents to rely on in Canada, there is even less clarity. In the absence of definitive legal rulings, most Canadian institutions, including TMU, advise against inputting copyrighted material into public LLMs. Complicating matters is the fact that the majority of electronic scholarly publications in academic libraries are licensed content, and these licenses come with their own restrictions above and sometimes beyond copyright. Open access material is an exception; generally it is understood that openly licensed content may be used in LLMs. Other institutions have gone further and suggested that vetted tools within an institution’s “walled garden” are appropriate for use with copyrighted content up to a prescribed data classification standard. The complexities of copyright, fair dealing, as well as terms of vendor license agreements makes it challenging to advise students and researchers on when and what is permitted. 

The capabilities of AI technologies go beyond the generation of text and include the generation of images and video. Caution must be exercised when integrating the use of AI tools into the production of creative works. Creators must be transparent and disclose AI use in any creative works and refrain from presenting creative works generated wholly or in part by AI tools as their sole work.

AI technologies are also having downstream effects on the scholarly ecosystem which must be monitored.  For instance, the use of AI tools in scholarly writing is reportedly increasing the volume of published research. While this may seem positive, there is a concern that it comes at the expense of quality. LLMs could amplify the "worst incentives of academia," such as the pressure to publish frequently, leading to a glut of low-quality, AI-generated papers.¹⁵ This increase in volume directly impacts the attribution of ideas and the fundamental norms of scholarly culture. The core of academia rests on the acknowledgment of sources and the proper citation of ideas. However, AI models "ingest" vast amounts of text, and the knowledge they produce is a synthesis of countless sources, making it difficult (if not impossible) to trace the origin of a specific idea. If AI models begin to extensively reuse and synthesize information without proper citation, traditional citation counts could become artificially depressed. As the Scholarly Kitchen¹⁶ notes, we may need to develop new "alternative impact indicators" to account for AI reuse, ensuring that the influence and importance of a work are accurately measured even if it is not formally cited in the traditional sense.

Similarly, there are reports of a dramatic increase in grant applications. The National Institutes of Health (NIH) has recently implemented a cap of six grant applications that may be submitted by a principal investigator per year. This is due to an increase in applications that have created strains on the application review process. (NIH policy (external link) ).

Back to top ↑

Learning and teaching

AI technologies in higher education, in many ways, present their greatest challenges for adaptation in the learning environment. As discussed above, AI has had its most immediate effects on the assessment of learning and academic integrity. Across the university, faculty and contract lecturers have been confronting the challenges with assessing student learning in the context of student AI use in the submission of coursework. Further to this challenge, students are concerned about instructor uses of AI, suspicious that course materials have been generated with AI tools and that their work is being assessed by AI technologies without disclosure and sufficient human oversight.¹⁷ This combination of challenges undermines trust within the classroom between students and instructors and necessitates clear articulation of expectations and transparency about AI use in the classroom. Furthermore, both instructors and students share concerns about cognitive offloading and intellectual atrophy as a result of overreliance on AI tools.

Upholding academic integrity is, perhaps, the primary challenge concerning AI in learning and teaching. This view is broadly shared among faculty and contract lecturers across the university in all disciplines. The Academic Integrity Office (AIO) has seen both an increase in the number of cases of academic misconduct involving inappropriate uses of AI and a transformation of the academic misconduct ecosystem itself.¹⁸ Previous forms of academic misconduct (plagiarism, contract cheating, etc.) have quickly been supplanted by AI misuse. The core challenge is identifying assessed coursework that does not represent the knowledge, understanding, and performance of the student. AI detection software remains unreliable, and the AIO continues to advise against the use of such technologies for the detection of academic misconduct. For these reasons, the task force has identified any form of assessed or graded coursework or program requirements to be high risk when using AI. Instructors should use scaffolded approaches to learning when integrating AI use in assessed work and ensure they are using a variety of methods for assessment security.¹⁹ All instructors should include clear guidelines and expectations concerning AI use in course outlines and make use of CELT resources for this purpose.

The task force sees a need for expanded guidelines on instructor uses of AI in courses to improve transparency and build trust with students. AI tools can be useful in creating course materials, making those materials more accessible, and reducing the administrative burden of faculty/contract lecturers. However, any use of AI tools in courses should be disclosed to students in a transparent manner. This disclosure also creates the opportunity for instructors and students to engage in dialogue about academic integrity, ethics, and trust.²⁰

The use of AI tools in the assessment of student learning is a high risk activity. Current TMU guidelines advise against the use of AI in grading. There are significant risks associated with this activity and uncertainties about the efficacy and effects of AI on the instructor-student relationship. We advise that the university continue to monitor the scholarship on this issue and support a range of pedagogical research on the effects of AI on learning and teaching.

While AI tools have the capacity to support student learning they also create the risk of undermining learning. Across the TMU community there is shared concern among instructors and students about the long-term effects of AI use on learning. Both students and faculty/contract lecturers see risk with overdependence on AI tools for cognitive offloading that may result in intellectual atrophy. Use of AI tools in learning can also result in bypassing foundational knowledge students require to gain competency in their disciplines. As such, instructors should communicate with their students about these potential risks and assess the potential effects of AI use on student achievement of course- and program-level learning outcomes. This may result in the integration of AI skills and competencies into courses to guide student learning. It may also result in the considered prohibition of AI tools to protect and ensure student learning.

Back to top ↑

Student experience

AI technologies present a range of challenges across the student experience at TMU. These challenges intersect with learning and teaching, SRC training, and operations. The task force identified challenges relating to advising, accessibility and accommodations, mental health and well-being, and access to other services. There are also considerable data privacy concerns to consider when it comes to student information. In community consultations, there was optimism about the potential for adopting AI systems to improve student services, but many in the community saw a need to ensure that there is always a human touch in the services TMU provides to students.

Although current Senate policy provides clarity on permissible uses of AI in learning much is left to the discretion of faculty/contract lecturers to delineate guidelines for individual courses. There is a need to improve communication with students about expectations and appropriate uses of AI in learning. Furthermore, students have raised concerns about unauthorized uses of AI detection tools and false suspicions of academic misconduct involving AI. As the university community continues to adapt to an AI-influenced learning environment it will be important to continue to monitor and address these concerns to ensure that TMU follows its principles of fairness and natural justice when it comes to the maintenance of academic integrity.

As with other areas with potential for improved service through the deployment of AI tools, student services may see benefits and risks with AI use. This is especially true in the areas of academic advising, accessibility and accommodations, and mental health and well being services. While AI chatbots may be useful for triaging low level inquiries and freeing up staff time for more meaningful high-level interactions with students, other student-facing services may not be suitable for the use of AI technologies. Factual errors and inaccuracies create significant risk for the use of AI automations and chatbots for academic advising. Academic accommodations and accessibility services also rely upon the provision of timely, accurate advice and guidance. The task force also draws particular attention to the high risk area of the use of AI for mental health and well-being services. Although it is recognized that significant numbers of students currently use AI chatbots as informal mental health aids, this behaviour can be risky and inadvisable. TMU will need to continue to devote attention to these mental health risks and provide students with appropriate support and guidance.

All university operations are subject to privacy laws and regulations and, of course, student information can be some of the most sensitive private data. Therefore, it will be especially important for the university to safeguard that data, and ensure its operations are compliant with regulations. The protection of student data can be aided by consistent use of institutional software and services and strong data privacy protections in the procurement and adoption of any AI technologies.

Back to top ↑

Administration and operations

Institutional adaptation to AI technologies will involve addressing significant and critical operational and technical challenges. These include challenges with data and integration, cybersecurity vulnerabilities, and data privacy risks. Additionally, there are several other challenges to consider in this realm such as shadow AI proliferation, training and AI competency gaps, software-as-a-service feature governance, client side AI interactions, cost controls, and vendor lock-in.

The adoption of any new AI tools will inevitably involve complicated data and integration hurdles. The complex technical and financial requirements of operationalizing selected tools via API integration with legacy ERP systems will require careful implementation to avoid exposing data perimeters. There is also inherent risk of introducing security vulnerabilities, dependencies, or unvetted scripts through unreviewed AI-assisted code generation. Rigorous cyber security review and protocols will be essential to the safe implementation of AI systems at the university. And strict data privacy reviews and processes will be essential to ensuring that TMU continues to meet its statutory and regulatory privacy obligations.

As TMU integrates AI technologies into its operations, it will be important to manage the administrative risk of staff utilizing unauthorized personal AI subscriptions for institutional work, leading to data leakage and introducing security vulnerability across unmanaged jurisdiction. Some of these risks can be avoided through comprehensive training and up-skilling for staff in data privacy and security. 

Another major operational challenge to consider is the need to determine how to govern, monitor, and selectively enable or disable automated AI features embedded inside vendor supplied SaaS platforms within the universities ecosystem. For example, Google Workspace offers a variety of AI integrations (in Gmail, Docs, Slides, Sheets, Calendar), some of which could pose significant risk of data leakage. Other vendor systems used across the university (library databases, for example), may automatically activate AI features without notification or providing choice. Finally, TMU will need to establish clear guidelines for professional units on how to guide, interact with, or evaluate client groups who are opaquely or transparently using AI to construct requests or interact with institutional teams. AI technologies present a series of risks in highly sensitive operational activities and environments including risks for data privacy, cybersecurity, and institutional reputation. Administrative units will have to determine the boundaries of appropriate, permissible uses of AI, depending on their operational functions at the university. For example, University Relations may need to consider a different approach to the use of AI for drafting text or creating images than Human Resources or Finance. Guidelines will need to be crafted to match the operational function of different units.

Back to top ↑